LVLMs and Humans Ground Differently in Referential Communication

TL;DR

This paper investigates the limitations of large vision-language models (LVLMs) in referential communication tasks, highlighting their inability to model common ground effectively compared to humans.

Contribution

It introduces a new experimental dataset and analysis tools to evaluate LVLMs' interactive referential communication skills against human performance.

Findings

LVLMs struggle to generate and resolve referring expressions interactively.

Humans outperform LVLMs in communication efficiency and accuracy.

The released dataset enables further research on AI-human collaborative language use.

Abstract

For generative AI agents to partner effectively with human users, the ability to accurately predict human intent is critical. But this ability to collaborate remains limited by a critical deficit: an inability to model common ground. We present a referential communication experiment with a factorial design involving director-matcher pairs (human-human, human-AI, AI-human, and AI-AI) that interact with multiple turns in repeated rounds to match pictures of objects not associated with any obvious lexicalized labels. We show that LVLMs cannot interactively generate and resolve referring expressions in a way that enables smooth communication, a crucial skill that underlies human language use. We release our corpus of 356 dialogues (89 pairs over 4 rounds each) along with the online pipeline for data collection and the tools for analyzing accuracy, efficiency, and lexical overlap.