Humility as a Dimension of Trusted AI
Not all model predictions are made with the same level of confidence. A trustworthy AI system knows when to be humble. Find out what it means for a prediction to be uncertain and what you can do about it.
An AI Prediction Is Fundamentally Probabilistic
Contextualizing how confident a prediction is in production enables more informed decision-making. An AI prediction is fundamentally probabilistic. There are various decision-making spaces where we are already accustomed to basing our decisions on probabilities, such as whether to bring a jacket when there’s a 40% chance of rain, betting odds on sports events, and political contests. In AI, unlike some of these examples, it is possible to explicitly and deterministically identify situations where a model’s prediction will have reduced confidence and to leverage that knowledge to make a better or safer decision. Recognizing and admitting uncertainty is a major step in establishing trust.
What Does It Mean for an AI Prediction to Have Less Confidence?
In an AI system, there are a couple ways to understand prediction confidence. Prediction intervals can be calculated and describe with a defined confidence the likelihood the actual value lies within a given range about a prediction. This interval can be highly informative; the narrower it is, the more confident the given prediction is. It also enables an understanding of best- and worst-case scenarios, at the highest and lowest values of the interval’s range.
In a classification setting, a prediction is based on a class probability. This provides an alternate understanding of confidence. For example in binary classification, the raw class probability will be a value somewhere between 0 and 1. The distribution of class probabilities will determine the classification threshold, over which a value of 1 will be assigned. But a simpler understanding of confidence can come from assessing if a value is very near or far from the classification threshold. Additionally, with access to a prediction interval, if the interval lies entirely to one side of a bound, you can more quantitatively assert the given confidence that the actual value is the predicted label.
Under What Circumstances Will a Prediction Be Less Confident or Certain?
A prediction might be less certain when confronting data measurably dissimilar from the data it was trained on. That might mean an outlier was input into one or more of the features. It might also mean a value the model has never seen before, such as a new categorical level, or only rarely seen was input. It is possible to identify a predicted value itself as an outlier from the predictions made in training or previously. These are conditions, beyond prediction intervals, that provide insight into the potential confidence of a model prediction. In DataRobot’s Humble AI, you can assign conditions like these as triggers to models in production with planned interventions automatically taking place in response.
What Kinds of Interventions Are Needed When a Prediction Is Uncertain?
A range of interventions is possible, along a spectrum from minimally disruptive to a full manual intervention into the decision-making process. The least disruptive intervention is to simply log and monitor uncertain predictions, including their triggering conditions. This log might provide insight into improvements that you can make to the model itself or its implementation and use. A user could be warned that the prediction is uncertain. Additionally, the system can return a default “safe” value instead of the original prediction’s output. At the most disruptive, an error can be returned, and/or a manual human operator alerted to intercede.
Humility is Just a Piece of the Puzzle
Humility that harmonizes the use of AI with your business as a whole is just one of the dimensions of trustworthy AI operations. The full list includes the following:
