GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers

TL;DR

GIFT is a framework that generates global, faithful, and human-readable textual explanations for vision classifiers by translating local counterfactuals into natural language and verifying their causal effect.

Contribution

GIFT introduces a novel method combining local visual counterfactuals, vision-language models, and causal verification to produce globally interpretable textual explanations for vision models.

Findings

Reveals meaningful classification rules across datasets

Identifies biases and latent concepts in vision models

Ensures explanations are faithful through causal verification

Abstract

Understanding the decision processes of deep vision models is essential for their safe and trustworthy deployment in real-world settings. Existing explainability approaches, such as saliency maps or concept-based analyses, often suffer from limited faithfulness, local scope, or ambiguous semantics. We introduce GIFT, a post-hoc framework that aims to derive Global, Interpretable, Faithful, and Textual explanations for vision classifiers. GIFT begins by generating a large set of faithful, local visual counterfactuals, then employs vision-language models to translate these counterfactuals into natural-language descriptions of visual changes. These local explanations are aggregated by a large language model into concise, human-readable hypotheses about the model's global decision rules. Crucially, GIFT includes a verification stage that quantitatively assesses the causal effect of each proposed explanation by performing image-based interventions, ensuring that the final textual explanations remain faithful to the model's true reasoning process. Across diverse datasets, including the synthetic CLEVR benchmark, the real-world CelebA faces, and the complex BDD driving scenes, GIFT reveals not only meaningful classification rules but also unexpected biases and latent concepts driving model behavior. Altogether, GIFT bridges the gap between local counterfactual reasoning and global interpretability, offering a principled approach to causally grounded textual explanations for vision models.