"Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations

TL;DR

This paper investigates the uncertainty inherent in LIME explanations, identifying sources of randomness and variation, and demonstrates how this uncertainty affects trust in model interpretations across different datasets.

Contribution

It reveals and analyzes the sources of uncertainty in LIME explanations, highlighting their impact on interpretability and trustworthiness of machine learning models.

Findings

Uncertainty arises from sampling randomness and data point variation.

Uncertainty exists even in highly accurate models.

Empirical analysis on synthetic and real datasets supports the findings.

Abstract

Methods for interpreting machine learning black-box models increase the outcomes' transparency and in turn generates insight into the reliability and fairness of the algorithms. However, the interpretations themselves could contain significant uncertainty that undermines the trust in the outcomes and raises concern about the model's reliability. Focusing on the method "Local Interpretable Model-agnostic Explanations" (LIME), we demonstrate the presence of two sources of uncertainty, namely the randomness in its sampling procedure and the variation of interpretation quality across different input data points. Such uncertainty is present even in models with high training and test accuracy. We apply LIME to synthetic data and two public data sets, text classification in 20 Newsgroup and recidivism risk-scoring in COMPAS, to support our argument.