Auditing Local Explanations is Hard

TL;DR

This paper demonstrates that auditing local explanations in machine learning models is inherently difficult, especially in high-dimensional settings, due to the high query complexity needed for verification.

Contribution

It introduces an auditing framework for local explanations, establishes bounds on query complexity, and highlights the importance of explanation locality in verification.

Findings

Auditing local explanations can require a large number of queries.

High-dimensional models significantly increase the difficulty of verification.

The locality of explanations is a crucial factor in their auditability.

Abstract

In sensitive contexts, providers of machine learning algorithms are increasingly required to give explanations for their algorithms' decisions. However, explanation receivers might not trust the provider, who potentially could output misleading or manipulated explanations. In this work, we investigate an auditing framework in which a third-party auditor or a collective of users attempts to sanity-check explanations: they can query model decisions and the corresponding local explanations, pool all the information received, and then check for basic consistency properties. We prove upper and lower bounds on the amount of queries that are needed for an auditor to succeed within this framework. Our results show that successful auditing requires a potentially exorbitant number of queries -- particularly in high dimensional cases. Our analysis also reveals that a key property is the ``locality'' of the provided explanations -- a quantity that so far has not been paid much attention to in the explainability literature. Looking forward, our results suggest that for complex high-dimensional settings, merely providing a pointwise prediction and explanation could be insufficient, as there is no way for the users to verify that the provided explanations are not completely made-up.