Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

TL;DR

This paper investigates whether vision-language models can recognize their own uncertainty and request clarification in reference games, highlighting current limitations and the potential of such tasks for testing model interaction capabilities.

Contribution

It introduces reference games as a testbed for assessing model uncertainty and clarification behavior, providing empirical evaluation of vision-language models in this context.

Findings

Models often fail to recognize their own uncertainty.

Models struggle to generate appropriate clarification requests.

Reference games are effective for testing interaction qualities.

Abstract

In human conversation, both interlocutors play an active role in maintaining mutual understanding. When listeners are uncertain about what speakers mean, for example, they can request clarification. It is an open question for language models whether they can assume a similar listener role, recognizing and expressing their own uncertainty through clarification. We argue that reference games are a suitable testbed to approach this question as they are controlled, self-contained, and make clarification needs explicit and measurable. To test this, we evaluate three vision-language models comparing a baseline reference resolution task to an experiment where the models are instructed to request clarification when uncertain. The results suggest that even in such simple tasks, models often struggle to recognize internal uncertainty and translate it into adequate clarification behavior. This demonstrates the value of reference games as testbeds for interaction qualities of (vision and) language models.