The Query Channel: Information-Theoretic Limits of Masking-Based Explanations

TL;DR

This paper models masking-based explanation methods as communication channels, deriving theoretical limits on explanation accuracy and comparing practical algorithms against these limits using information theory.

Contribution

It introduces an information-theoretic framework for understanding explanation complexity and establishes capacity bounds for reliable feature importance recovery.

Findings

Exact recovery probability converges to one if explanation rate exceeds capacity.

Sparse maximum-likelihood decoder achieves reliable recovery below capacity.

Standard convex surrogates like LIME and KernelSHAP often fail within certain query budgets.

Abstract

Masking-based post-hoc explanation methods, such as KernelSHAP and LIME, estimate local feature importance by querying a black-box model under randomized perturbations. This paper formulates this procedure as communication over a query channel, where the latent explanation acts as a message and each masked evaluation is a channel use. Within this framework, the complexity of the explanation is captured by the entropy of the hypothesis class, while the query interface supplies information at a rate determined by an identification capacity per query. We derive a strong converse showing that, if the explanation rate exceeds this capacity, the probability of exact recovery necessarily converges to one in error for any sequence of explainers and decoders. We also prove an achievability result establishing that a sparse maximum-likelihood decoder attains reliable recovery when the rate lies below capacity. A Monte Carlo estimator of mutual information yields a non-asymptotic query benchmark that we use to compare optimal decoding with Lasso- and OLS-based procedures that mirror LIME and KernelSHAP. Experiments reveal a range of query budgets where information theory permits reliable explanations but standard convex surrogates still fail. Finally, we interpret super-pixel resolution and tokenization for neural language models as a source-coding choice that sets the entropy of the explanation and show how Gaussian noise and nonlinear curvature degrade the query channel, induce waterfall and error-floor behavior, and render high-resolution explanations unattainable.