LLM-based Hierarchical Concept Decomposition for Interpretable Fine-Grained Image Classification

TL;DR

This paper introduces Hi-CoDe, a hierarchical concept decomposition framework that enhances interpretability in fine-grained image classification by structuring visual concepts and using simple classifiers, maintaining high accuracy.

Contribution

The novel Hi-CoDe framework combines structured concept hierarchies with linear classifiers, improving interpretability without sacrificing performance in image classification.

Findings

Achieves state-of-the-art interpretability in fine-grained classification.

Provides clear insights into decision-making processes.

Maintains competitive accuracy with complex models.

Abstract

(Renyi Qu's Master's Thesis) Recent advancements in interpretable models for vision-language tasks have achieved competitive performance; however, their interpretability often suffers due to the reliance on unstructured text outputs from large language models (LLMs). This introduces randomness and compromises both transparency and reliability, which are essential for addressing safety issues in AI systems. We introduce \texttt{Hi-CoDe} (Hierarchical Concept Decomposition), a novel framework designed to enhance model interpretability through structured concept analysis. Our approach consists of two main components: (1) We use GPT-4 to decompose an input image into a structured hierarchy of visual concepts, thereby forming a visual concept tree. (2) We then employ an ensemble of simple linear classifiers that operate on concept-specific features derived from CLIP to perform classification. Our approach not only aligns with the performance of state-of-the-art models but also advances transparency by providing clear insights into the decision-making process and highlighting the importance of various concepts. This allows for a detailed analysis of potential failure modes and improves model compactness, therefore setting a new benchmark in interpretability without compromising the accuracy.