Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inference

TL;DR

This paper introduces a new benchmark and a contrastive inference framework for explainable object recognition that handles multiple rationales, improving accuracy and interpretability without training.

Contribution

It proposes a multi-rationale dataset and a contrastive conditional inference method that models relationships among image features, labels, and rationales, achieving state-of-the-art results.

Findings

Achieved state-of-the-art performance on the benchmark.

Demonstrated strong zero-shot classification accuracy.

Produced high-quality rationales for explanations.

Abstract

Explainable object recognition using vision-language models such as CLIP involves predicting accurate category labels supported by rationales that justify the decision-making process. Existing methods typically rely on prompt-based conditioning, which suffers from limitations in CLIP's text encoder and provides weak conditioning on explanatory structures. Additionally, prior datasets are often restricted to single, and frequently noisy, rationales that fail to capture the full diversity of discriminative image features. In this work, we introduce a multi-rationale explainable object recognition benchmark comprising datasets in which each image is annotated with multiple ground-truth rationales, along with evaluation metrics designed to offer a more comprehensive representation of the task. To overcome the limitations of previous approaches, we propose a contrastive conditional inference (CCI) framework that explicitly models the probabilistic relationships among image embeddings, category labels, and rationales. Without requiring any training, our framework enables more effective conditioning on rationales to predict accurate object categories. Our approach achieves state-of-the-art results on the multi-rationale explainable object recognition benchmark, including strong zero-shot performance, and sets a new standard for both classification accuracy and rationale quality. Together with the benchmark, this work provides a more complete framework for evaluating future models in explainable object recognition. The code will be made available online.