Evaluating Attribute Comprehension in Large Vision-Language Models

TL;DR

This paper assesses how well large vision-language models understand object attributes, focusing on recognition and hierarchy, revealing strengths and limitations in their fine-grained visual comprehension abilities.

Contribution

It introduces a comprehensive evaluation framework for attribute comprehension in vision-language models, highlighting the impact of fine-tuning data and interaction types on their understanding.

Findings

Models excel at attribute recognition but have limited hierarchical understanding.

Image-text matching outperforms visual question answering in attribute comprehension.

Caption attribute information significantly influences fine-tuning effectiveness.

Abstract

Currently, large vision-language models have gained promising progress on many downstream tasks. However, they still suffer many challenges in fine-grained visual understanding tasks, such as object attribute comprehension. Besides, there have been growing efforts on the evaluations of large vision-language models, but lack of in-depth study of attribute comprehension and the visual language fine-tuning process. In this paper, we propose to evaluate the attribute comprehension ability of large vision-language models from two perspectives: attribute recognition and attribute hierarchy understanding. We evaluate three vision-language interactions, including visual question answering, image-text matching, and image-text cosine similarity. Furthermore, we explore the factors affecting attribute comprehension during fine-tuning. Through a series of quantitative and qualitative experiments, we introduce three main findings: (1) Large vision-language models possess good attribute recognition ability, but their hierarchical understanding ability is relatively limited. (2) Compared to ITC, ITM exhibits superior capability in capturing finer details, making it more suitable for attribute understanding tasks. (3) The attribute information in the captions used for fine-tuning plays a crucial role in attribute understanding. We hope this work can help guide future progress in fine-grained visual understanding of large vision-language models.