LICO: Explainable Models with Language-Image Consistency

TL;DR

LICO introduces a novel explainable image classification approach that aligns language prompts with visual features, producing more accurate saliency maps and improving classification performance without extra inference costs.

Contribution

The paper proposes LICO, a new model that correlates linguistic prompts with visual features using coarse-to-fine alignment and optimal transport, enhancing interpretability and accuracy.

Findings

LICO generates more explainable attention maps than existing methods.

LICO improves classification accuracy without additional inference overhead.

Experimental results on eight datasets validate LICO's effectiveness.

Abstract

Interpreting the decisions of deep learning models has been actively studied since the explosion of deep neural networks. One of the most convincing interpretation approaches is salience-based visual interpretation, such as Grad-CAM, where the generation of attention maps depends merely on categorical labels. Although existing interpretation methods can provide explainable decision clues, they often yield partial correspondence between image and saliency maps due to the limited discriminative information from one-hot labels. This paper develops a Language-Image COnsistency model for explainable image classification, termed LICO, by correlating learnable linguistic prompts with corresponding visual features in a coarse-to-fine manner. Specifically, we first establish a coarse global manifold structure alignment by minimizing the distance between the distributions of image and language features. We then achieve fine-grained saliency maps by applying optimal transport (OT) theory to assign local feature maps with class-specific prompts. Extensive experimental results on eight benchmark datasets demonstrate that the proposed LICO achieves a significant improvement in generating more explainable attention maps in conjunction with existing interpretation methods such as Grad-CAM. Remarkably, LICO improves the classification performance of existing models without introducing any computational overhead during inference. Source code is made available at https://github.com/ymLeiFDU/LICO.