Evaluating the encoding competence of visual language models using uncommon actions

TL;DR

This paper introduces UAIT, a benchmark dataset for evaluating visual language models' understanding of uncommon-sense actions, revealing their limitations and potential for improvement in semantic reasoning.

Contribution

The paper presents UAIT, a novel dataset for testing VLMs on uncommon-sense actions, and provides insights into their semantic reasoning weaknesses and adaptation potential.

Findings

Models perform worse than humans on semantic judgment tasks.

Fine-tuning improves model accuracy in uncommon-sense reasoning.

The dataset reveals key weaknesses in current VLMs' understanding.

Abstract

We propose UAIT (Uncommon-sense Action Image-Text) dataset, a new evaluation benchmark designed to test the semantic understanding ability of visual language models (VLMs) in uncommon-sense action scenes. Unlike previous datasets that focus on common visual scenes with statistical frequency advantages, UAIT challenges models with grammatically reasonable but semantically counter-common sense image-text pairs. Such tasks require models to go beyond superficial pattern recognition and demonstrate a deep understanding of agent-patient relationships and physical feasibility. To build UAIT, we designed a semi-automated process to synthesize high-quality uncommon-sense image-text samples using large language models, few-shot prompt engineering, and text-to-image generation. Each sample is accompanied by a carefully designed multiple-choice question to test the model's competence in fine-grained reasoning. We evaluate multiple state-of-the-art visual language models and compare them with models based on contrastive learning. Experiments show that all models perform significantly worse than humans in semantic judgment, especially in distinguishing grammatical correctness from semantic rationality. Further experiments show that even the lightweight model can improve its accuracy after fine-tuning, demonstrating the great potential of directional adaptation. This study not only reveals the key weaknesses of VLMs, but also provides diagnostic tools and research directions for the development of robust models with real visual semantic reasoning capabilities.