What's meant by explainable model: A Scoping Review

TL;DR

This paper reviews how the term 'explainable model' is used in AI literature, revealing most works claiming explainability do not evaluate the quality or suitability of their explanation methods, especially post-hoc techniques.

Contribution

It highlights the prevalent misuse of the term 'explainable' in AI research and emphasizes the need for proper evaluation of explanation methods within specific applications.

Findings

81% of papers claiming explainability do not evaluate their explanation methods

Post-hoc explanation methods often lack proper assessment of quality

Explainability claims are frequently made without sufficient validation

Abstract

We often see the term explainable in the titles of papers that describe applications based on artificial intelligence (AI). However, the literature in explainable artificial intelligence (XAI) indicates that explanations in XAI are application- and domain-specific, hence requiring evaluation whenever they are employed to explain a model that makes decisions for a specific application problem. Additionally, the literature reveals that the performance of post-hoc methods, particularly feature attribution methods, varies substantially hinting that they do not represent a solution to AI explainability. Therefore, when using XAI methods, the quality and suitability of their information outputs should be evaluated within the specific application. For these reasons, we used a scoping review methodology to investigate papers that apply AI models and adopt methods to generate post-hoc explanations while referring to said models as explainable. This paper investigates whether the term explainable model is adopted by authors under the assumption that incorporating a post-hoc XAI method suffices to characterize a model as explainable. To inspect this problem, our review analyzes whether these papers conducted evaluations. We found that 81% of the application papers that refer to their approaches as an explainable model do not conduct any form of evaluation on the XAI method they used.