Large Language Models Facilitate Vision Reflection in Image Classification

TL;DR

This paper reveals how large multimodal models can improve image recognition accuracy and interpretability through vision reflection, leveraging textual concepts and a training-free connector, offering new insights into explainability.

Contribution

It introduces novel methods for enhancing vision-language models' accuracy and interpretability using vision reflection and textual reasoning without extensive training.

Findings

Prompting LMMs improves recognition accuracy on ImageNet.

Vision-language connector maps visual features into textual concepts.

A training-free connector enhances fine-grained recognition.

Abstract

This paper presents several novel findings on the explainability of vision reflection in large multimodal models (LMMs). First, we show that prompting an LMM to verify the prediction of a specialized vision model can improve recognition accuracy, even on benchmarks like ImageNet, despite prior evidence that LMMs typically underperform dedicated vision encoders. Second, we analyze the internal behavior of vision reflection and find that the vision-language connector maps visual features into explicit textual concepts, allowing the language model to reason about prediction plausibility using commonsense knowledge. We further observe that replacing a large number of vision tokens with only a few text tokens still enables LLaVA to generate similar answers, suggesting that LMMs may rely primarily on a compact set of distilled textual representations rather than raw vision features. Third, we show that a training-free connector can enhance LMM performance in fine-grained recognition tasks, without extensive feature-alignment training. Together, these findings offer new insights into the explainability of vision-language models and suggest that vision reflection is a promising strategy for achieving robust and interpretable visual recognition.