Model Monitoring
What Is Model Monitoring?
Model monitoring is the close tracking of the performance of ML models in production so that production and AI teams can identify potential issues before they impact the business. A robust MLOps infrastructure should be able to proactively monitor for service health (that is, its availability),assess data relevance (the variance between the data models were trained on and the live data it is scoring against across all features), model performance (model accuracy), trust elements such as fairness and bias, and of course, business impact.
Why Is Model Monitoring Important?
The predictive performance of a model typically starts to diminish as soon as it’s deployed. For example, someone might be making live predictions on a dataset with customer data, but the customer’s behavioral patterns might have changed due to an economic crisis, market volatility, natural disaster, or even the weather. Models trained on older data that no longer represents the current reality might not just be inaccurate, but irrelevant, leaving the prediction results meaningless or even harmful. Without dedicated production model monitoring, the user or business owner cannot know or be able to detect when this happens. If model accuracy starts to decline without detection, the results can impact a business, expose it to risk, and destroy user trust.
Model Monitoring and DataRobot
DataRobot machine learning operations (MLOps) is a key pillar of the DataRobot enterprise AI platform. It provides a scalable and governed means to rapidly deploy and manage machine learning applications in production environments. The four pillars of DataRobot MLOps are model monitoring, simplified model deployment, production lifecycle management, and production model governance. With DataRobot MLOps, models built on any machine learning platform or in practically any language can be deployed, tested, and seamlessly updated in production on any runtime environment and managed from a single location. There are controls in place to natively integrate with most access, log, and resource management services to seamlessly integrate with any production environment and for legal and regulatory compliance.
Remote Model Monitoring Architecture
The architecture includes three layers:
MLOps Library
The MLOps Library can be invoked by a model’s scoring code to track service, drift, and accuracy metrics. The library is responsible for capturing metrics and posting them to the configured channel. The library supports common languages, such as Java, Python, and R.
The MLOps Library can be used to report to a highly scalable channel, such as Google Cloud Pub/Sub or Amazon SQS, for scalable, near real-time monitoring. Or it can be used to report metrics outside of the prediction path.
Channel
There are multiple channels available to pass metrics captured by the MLOps Library as the model makes predictions. Passing metrics over a channel instead of directly gives you more configuration options and provides support for disconnected models that do not have network connectivity to the MLOps application.
The following channels are available:
● File system
● RabbitMQ
● Amazon SQS
● Google Pub/Sub
MLOps Agent
DataRobot is the first in the industry to use agents to collect and analyze monitoring statistics from models running practically anywhere. These statistics are available in the customer’s MLOps center of excellence (CoE). The MLOps Agent is a lightweight software program designed to run on remote hosts that monitors an application’s behavior, including its events, performance, and availability. The DataRobot MLOps Agent supports any model, written in any language, deployed in any environment, including:
● Models developed with open-source languages and libraries and deployed outside of DataRobot (Amazon Web Services, Microsoft Azure, Google Cloud Platform, on-premise).
● Models running on distributed compute platforms, such as Spark.
● DataRobot models downloaded and deployed on organization infrastructure.
● Stored procedures running in modern databases.
The MLOps Agent watches the channel for updated metrics, collects them, and passes them to the MLOps application. This capability provides instant visibility into the performance of models that are running anywhere in your environment. The MLOps Agent can be deployed anywhere it has access to the channel—in the prediction environment or on the edge. Advanced capabilities may include features that have been built into the MLOps solution to help customers increase their trust in their models in production even more.
For example, in AI the principle of humility dictates that models should be able to inform not only when predictions are possibly losing accuracy or relevance, but also when they’re not confident in the quality of the fairness of their predictions. When this information is captured and sent to a centralized server, it’s much easier to detect and diagnose issues occurring in any remote application. DataRobot MLOps and its model monitoring capabilities make it possible for organizations to run their models in their preferred infrastructure and quickly get metrics on any deployed model in a single, centralized environment.