Real-Time Drift Drill Down Simplifies Ad Hoc Drift Analysis

Data drift is a phenomenon that reflects natural changes in the world around us, such as shifts in consumer demand, economic fluctuation, or a force majeure. While changes in new data can threaten the performance of production models, data drift can be a strategic opportunity for your AI solution to quickly adapt to new patterns and maintain competitive advantage over not-so-quick competitors. The key, of course, is your response time: how quickly data drift can be analyzed and corrected. 

New in DataRobot AI Platform is a unique drift drill down capability to help you manage change and stay ahead of your competition. 

Change is inevitable. Growth is optional.

Drill Down into Drift for Rapid Model Diagnostics 

A key challenge in investigating data drift is the lack of details available to the user. Traditionally, drift is tracked for top features by comparing scoring data to training data. Drift can also be viewed over time to identify general drift trends. To dive deeper into the patterns and causes of drift, MLOps users need to be able to compare drift between two scoring data segments (in addition to the traditional comparison between scoring and training data), for any or all features, and across any specified time period. 

DataRobot MLOps users can now compare drift of selected features between two scoring segments of a model (or scoring and training segments), for any time period, and view contextual information such as prediction value over time to further support their investigation. 

As highlighted in the DataRobot interface above, the Data Drift tab is enhanced with a Drill Down section for users to visualize drift details. Users can configure their own display settings to select a model, date range of interest, and time granularity. This is important as data drift can look different at different time granularities; drift can happen at any time and at any rate. 

For example, if a model has been in production for a year with little drift, but has only begun drifting at an increasing rate in the last week, the overall drift view may not represent this imminent problem. Zooming into that last week will help the user understand how quickly data is drifting and whether or not it’s a cause for concern. 

“You might think that overall, the model’s features drifted relatively little in production, but in reality, the model’s drift statistics might be fluctuating quite a bit up and down. Or there might be a concerning trend beginning to develop over the past week that you want to keep an eye on. That insight requires looking at specific time slices. Granular time splits show you the true picture,” emphasized Brian Bell, Senior Director, Product Management who leads the DataRobot MLOps strategy.

Without the ability to zoom into granular time slices, differences in drift patterns may get lost in the overall analysis. The new DataRobot drift drill down capability allows data scientists to run quick sanity checks, investigate accelerating or decelerating patterns in drift, and control the level of granularity of the visuals.

DataRobot offers fast and intuitive drift drill down, as we focus on analyzing your data across different dimensions in real-time to answer data science questions. From our interface you can change the parameters of analysis and get to multiple insights quickly.

Rapid Product Development for a Fast-Changing Economy

The DataRobot drift drill down capability was inspired by a conversation with a bank as their data science team struggled with ad hoc drift analysis. Prior to using DataRobot, the customer was conducting tedious experimentation to track and investigate drift patterns. Their data science team did not have a straightforward way to ask targeted questions about changes in data over specified time periods. They needed to conduct drift analysis in real time to ensure the performance of deployed models.

The bank’s data science team saw value in the ability to conduct drift deep dive and answer critical questions within seconds.

The need for ad hoc drift deep dive is being felt by more and more organizations, especially as world economic conditions continue to impact models at an alarming rate. Patterns in data are changing faster than data science teams can keep up with, costing them time and visibility into deployments. Drift drill down solves this data science challenge so that organizations can maintain AI driven business results.

MLOps Is Vital for Enterprise AI

DataRobot MLOps offers a single place to deploy, monitor, manage, and govern models in production, regardless of how they were created or when and where they were deployed. Learn more about DataRobot MLOps. 

About the author

May Masoud

Data Scientist, DataRobot

May Masoud is a data scientist, AI advocate, and thought leader trained in classical Statistics and modern Machine Learning. At DataRobot she designs market strategy for the DataRobot AI Platform, helping global organizations derive measurable return on AI investments while maintaining enterprise governance and ethics.

May developed her technical foundation through degrees in Statistics and Economics, followed by a Master of Business Analytics from the Schulich School of Business. This cocktail of technical and business expertise has shaped May as an AI practitioner and a thought leader. May delivers Ethical AI and Democratizing AI keynotes and workshops for business and academic communities.

Meet May Masoud