Metonymy in vision models undermines attention-based interpretability

TL;DR

This paper reveals that modern vision transformers violate the locality assumption, causing intra-object leakage that undermines the interpretability of part-based reasoning methods.

Contribution

It demonstrates the presence of intra-object leakage in vision models, shows its impact on interpretability, and proposes a two-stage approach to mitigate this issue.

Findings

Modern pretrained vision transformers exhibit strong intra-object leakage.

Intra-object leakage compromises the faithfulness of attention-based interpretability methods.

A two-stage approach can prevent leakage and improve attribute-driven part discovery.

Abstract

Part-based reasoning is a classical strategy to make a computer vision model directly focus on the object parts that are relevant to the downstream task. In the context of deep learning, this also serves to improve by-design interpretability, often by using part-centric attention mechanisms on top of a latent image representation provided by a standard, black-box model. This approach is based on a locality assumption: that the latent representation of an object part encodes primarily information about the corresponding image region. In this work, we test this basic assumption, measuring intra-object leakage in vision models using part-based attribute annotations. Through a comprehensive experimental evaluation, we show that modern pretrained vision transformers violate the locality assumption and exhibit a strong intra-object leakage, in which each part encodes information from the whole object, a visual metonymy that compromises the faithfulness of attention-based interpretable-by-design methods for part-based reasoning, ultimately rendering them uninterpretable. In addition, we establish an upper bound using a two-stage approach that prevents leakage by design. We then show that this inherently disentangled feature extraction improves attribute-driven part discovery on a variety of tasks, confirming the practical impact of intra-object leakage. Our results uncover a neglected issue affecting the interpretability of part-based representations, such as those in CBMs relying on part-centric concepts, highlighting that two-stage approaches offer a promising way to mitigate it.