Probing Vision-Language Understanding through the Visual Entailment Task: promises and pitfalls

TL;DR

This paper critically examines the Visual Entailment task as a probe for vision-language understanding in multimodal models, revealing its strengths and limitations through extensive experiments and analysis.

Contribution

It provides a comprehensive evaluation of VE's effectiveness as a diagnostic tool and highlights factors influencing model performance, such as prompt design and visual information access.

Findings

Three-shot inference outperforms zero-shot baseline.

Additional examples can introduce noise, affecting performance.

Fine-tuning achieves 83.3% accuracy on e-SNLI-VE and produces meaningful explanations.

Abstract

This study investigates the extent to which the Visual Entailment (VE) task serves as a reliable probe of vision-language understanding in multimodal language models, using the LLaMA 3.2 11B Vision model as a test case. Beyond reporting performance metrics, we aim to interpret what these results reveal about the underlying possibilities and limitations of the VE task. We conduct a series of experiments across zero-shot, few-shot, and fine-tuning settings, exploring how factors such as prompt design, the number and order of in-context examples and access to visual information might affect VE performance. To further probe the reasoning processes of the model, we used explanation-based evaluations. Results indicate that three-shot inference outperforms the zero-shot baselines. However, additional examples introduce more noise than they provide benefits. Additionally, the order of the labels in the prompt is a critical factor that influences the predictions. In the absence of visual information, the model has a strong tendency to hallucinate and imagine content, raising questions about the model's over-reliance on linguistic priors. Fine-tuning yields strong results, achieving an accuracy of 83.3% on the e-SNLI-VE dataset and outperforming the state-of-the-art OFA-X model. Additionally, the explanation evaluation demonstrates that the fine-tuned model provides semantically meaningful explanations similar to those of humans, with a BERTScore F1-score of 89.2%. We do, however, find comparable BERTScore results in experiments with limited vision, questioning the visual grounding of this task. Overall, our results highlight both the utility and limitations of VE as a diagnostic task for vision-language understanding and point to directions for refining multimodal evaluation methods.