Does VLM Classification Benefit from LLM Description Semantics?

TL;DR

This paper investigates whether Large Language Model-generated descriptions genuinely enhance Vision-Language Model classification by analyzing their semantic contribution, proposing an evaluation scenario and a training-free selection method to improve accuracy and explainability.

Contribution

It introduces an evaluation scenario to distinguish true semantic benefits from noise effects and proposes a training-free method for selecting discriminative descriptions for VLM classification.

Findings

Descriptions with genuine semantics improve classification accuracy.

The proposed method outperforms baseline approaches across seven datasets.

Insights into explainability of description-based classification are provided.

Abstract

Accurately describing images with text is a foundation of explainable AI. Vision-Language Models (VLMs) like CLIP have recently addressed this by aligning images and texts in a shared embedding space, expressing semantic similarities between vision and language embeddings. VLM classification can be improved with descriptions generated by Large Language Models (LLMs). However, it is difficult to determine the contribution of actual description semantics, as the performance gain may also stem from a semantic-agnostic ensembling effect, where multiple modified text prompts act as a noisy test-time augmentation for the original one. We propose an alternative evaluation scenario to decide if a performance boost of LLM-generated descriptions is caused by such a noise augmentation effect or rather by genuine description semantics. The proposed scenario avoids noisy test-time augmentation and ensures that genuine, distinctive descriptions cause the performance boost. Furthermore, we propose a training-free method for selecting discriminative descriptions that work independently of classname-ensembling effects. Our approach identifies descriptions that effectively differentiate classes within a local CLIP label neighborhood, improving classification accuracy across seven datasets. Additionally, we provide insights into the explainability of description-based image classification with VLMs.