Can Vision-Language Models Infer Speaker's Ignorance? The Role of Visual and Linguistic Cues

TL;DR

This paper examines whether vision-language models can infer speaker ignorance using visual and linguistic cues, revealing that some models like Claude show emerging pragmatic reasoning abilities.

Contribution

It demonstrates how different VLMs process contextual cues for pragmatic inference, highlighting Claude's potential for integrating multiple cues for human-like reasoning.

Findings

Claude integrates visual and linguistic cues more effectively.

GPT and Gemini interpret cues literally without combining them.

Models treat cues independently, showing limited pragmatic reasoning.

Abstract

This study investigates whether vision-language models (VLMs) can perform pragmatic inference, focusing on ignorance implicatures, utterances that imply the speaker's lack of precise knowledge. To test this, we systematically manipulated contextual cues: the visually depicted situation (visual cue) and QUD-based linguistic prompts (linguistic cue). When only visual cues were provided, three state-of-the-art VLMs (GPT-4o, Gemini 1.5 Pro, and Claude 3.5 sonnet) produced interpretations largely based on the lexical meaning of the modified numerals. When linguistic cues were added to enhance contextual informativeness, Claude exhibited more human-like inference by integrating both types of contextual cues. In contrast, GPT and Gemini favored precise, literal interpretations. Although the influence of contextual cues increased, they treated each contextual cue independently and aligned them with semantic features rather than engaging in context-driven reasoning. These findings suggest that although the models differ in how they handle contextual cues, Claude's ability to combine multiple cues may signal emerging pragmatic competence in multimodal models.