How Understanding Forecast Uncertainty Resolves the Explainability Problem in Machine Learning Models

TL;DR

This paper clarifies that explainability issues in machine learning are due to forecast uncertainty at decision boundaries, and proposes focusing on forecast usability to determine when explanations are meaningful.

Contribution

It introduces a framework that links forecast uncertainty with explainability, emphasizing the importance of assessing forecast usability before applying local explanations.

Findings

High uncertainty at decision boundaries causes explanation instability.

Usable forecasts enable reliable local linear explanations.

Piecewise linear models like ReLU networks can have illusory explainability.

Abstract

For applications of machine learning in critical decisions, explainability is a primary concern, and often a regulatory requirement. Local linear methods for generating explanations, such as LIME and SHAP, have been criticized for being unstable near decision boundaries. In this paper, we explain that such concerns reflect a misunderstanding of the problem. The forecast uncertainty is high at decision boundaries, so consequently, the explanatory instability is high. The correct approach is to change the sequence of events and questions being asked. Nonlinear models can be highly predictive in some regions while having little or no predictability in others. Therefore, the first question is whether a usable forecast exists. When there is a forecast with low enough uncertainty to be useful, an explanation can be sought via a local linear approximation. In such cases, the explanatory instability is correspondingly low. When no usable forecast exists, the decision must fall to a simpler overall model such as traditional logistic regression. Additionally, these results show that some methods that purport to be explainable everywhere, such as ReLU networks or any piecewise linear model, have only an illusory explainability, because the forecast uncertainty at the segment boundaries is too high to be useful. Explaining an unusable forecast is pointless.