The Bouncer Problem: Challenges to Remote Explainability

TL;DR

This paper argues that remote explainability of machine learning models is fundamentally impossible because service providers can provide false explanations, as demonstrated through theoretical proofs and practical attack examples.

Contribution

The paper introduces the 'Bouncer Problem' analogy, proves the impossibility of truthful remote explanations, and demonstrates practical attacks that conceal discriminatory features.

Findings

Remote explanations can be manipulated to hide true decision reasons

Attacks on explanations are difficult to detect in practice

Remote explainability cannot guarantee transparency or truthfulness

Abstract

The concept of explainability is envisioned to satisfy society's demands for transparency on machine learning decisions. The concept is simple: like humans, algorithms should explain the rationale behind their decisions so that their fairness can be assessed. While this approach is promising in a local context (e.g. to explain a model during debugging at training time), we argue that this reasoning cannot simply be transposed in a remote context, where a trained model by a service provider is only accessible through its API. This is problematic as it constitutes precisely the target use-case requiring transparency from a societal perspective. Through an analogy with a club bouncer (which may provide untruthful explanations upon customer reject), we show that providing explanations cannot prevent a remote service from lying about the true reasons leading to its decisions. More precisely, we prove the impossibility of remote explainability for single explanations, by constructing an attack on explanations that hides discriminatory features to the querying user. We provide an example implementation of this attack. We then show that the probability that an observer spots the attack, using several explanations for attempting to find incoherences, is low in practical settings. This undermines the very concept of remote explainability in general.