Scoring Data
What does Scoring Data Mean?
Note: In data science, there are two types of scoring: model scoring and scoring data. This article is about the latter type.
In machine learning, scoring is the process of applying an algorithmic model built from a historical dataset to a new dataset in order to uncover practical insights that will help solve a business problem.
Model development is generally a two-stage process. The first stage is training and validation, during which you apply algorithms to data for which you know the outcomes to uncover patterns between its features and the target variable. The second stage is scoring, in which you apply the trained model to a new dataset. Then, the model returns outcomes in the form of probability scores for classification problems and estimated averages for regression problems. Finally, you deploy the trained model into a production application or use the insights it uncovers to improve business processes.
For example, to score a model meant to predict the likelihood of customer churn:
- Build a churn behavior model using a historical dataset that contains information on which customers churned and other information that you believe contributed to that outcome.
- Apply the model to existing customer data to produce a value, or “score,” that estimates their likelihood to churn.
Different ways to score models include:
- Batch scoring. Useful for when the model’s decisions don’t have to be implemented immediately. For example, a marketer may batch score a model on a list of purchased leads to determine which are most likely to buy their product.
- Real-time scoring. Useful when time is of the essence in realizing value from the model. For example, a bank needs a fraud model to score credit card transactions within milliseconds to quickly deny likely fraudulent transactions.
Scoring is also used to evaluate existing models. By training the model on historical data, using it to score other historical data for which you know the outcome, and comparing the scores to the known values, you determine how well the model performs.
Why is Scoring Important?
Scoring is a key component of understanding machine learning model outcomes and choosing the most accurate model that produces the most valuable insights. Once you have a model in production scoring new data, you’ll uncover insights that you can use to create business value.
Using the above example, the model scores identify which current customers are at a high risk of churning, enabling you to plan outreach or special offers to prevent that from happening.
Scoring + DataRobot
DataRobot’s Prediction Explanations feature has a great visualization of model output scores:
DataRobot shows the score in the second column from the left after the individual record ID.
In the above example, a hospital has built a classification model to determine the likelihood that a patient will be readmitted in 30 days or less. The model’s score for patient ID 9155 is 0.888. In other words, this patient had an 88.8% likelihood of being readmitted prior to the end of 30 days. The “Explanations” columns list the top factors that contributed to that probability score.
Using the model score, the hospital can take action to reduce the probability of readmissions, which might include delaying the discharge of patients that are similar to those with high readmission scores, resulting in better patient outcomes and a lower instance of the hospital being fined.
For information on how DataRobot handles scoring and deployment, see the Deployment wiki entry.