What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers

TL;DR

This paper introduces BiCAM, a bidirectional explanation method for Vision Transformers that captures both positive and negative contributions, improving interpretability and enabling lightweight adversarial detection across multiple datasets and models.

Contribution

BiCAM is the first CAM-based approach to preserve signed attributions, providing more complete explanations and a novel Positive-to-Negative Ratio for adversarial detection in Vision Transformers.

Findings

BiCAM improves localization and faithfulness in explanations.

It enables lightweight adversarial example detection.

The method generalizes across multiple ViT variants.

Abstract

Vision Transformers (ViTs) achieve strong performance in visual recognition, yet their decision-making remains difficult to interpret. We propose BiCAM, a bidirectional class activation mapping method that captures both supportive (positive) and suppressive (negative) contributions to model predictions. Unlike prior CAM-based approaches that discard negative signals, BiCAM preserves signed attributions to produce more complete and contrastive explanations. BiCAM further introduces a Positive-to-Negative Ratio (PNR) that summarizes attribution balance and enables lightweight detection of adversarial examples without retraining. Across ImageNet, VOC, and COCO, BiCAM improves localization and faithfulness while remaining computationally efficient. It generalizes to multiple ViT variants, including DeiT and Swin. These results suggest the importance of modeling both supportive and suppressive evidence for interpreting transformer-based vision models.