CLUE: Crossmodal disambiguation via Language-vision Understanding with attEntion

TL;DR

CLUE introduces a novel approach that transforms the internal attention mechanisms of vision-language models into explicit signals for ambiguity detection, enabling robots to better interpret human intentions and decide when to ask for clarification during interaction.

Contribution

The paper presents a method to convert cross-modal attention into an explicit ambiguity detection signal, improving interactive visual grounding in human-robot interaction.

Findings

Outperforms state-of-the-art methods with InViG supervision

Ambiguity detector surpasses prior baselines

Uses parameter-efficient fine-tuning with LoRA

Abstract

With the increasing integration of robots into daily life, human-robot interaction has become more complex and multifaceted. A critical component of this interaction is Interactive Visual Grounding (IVG), through which robots must interpret human intentions and resolve ambiguity. Existing IVG models generally lack a mechanism to determine when to ask clarification questions, as they implicitly rely on their learned representations. CLUE addresses this gap by converting the VLM's cross-modal attention into an explicit, spatially grounded signal for deciding when to ask. We extract text to image attention maps and pass them to a lightweight CNN to detect referential ambiguity, while a LoRA fine-tuned decoder conducts the dialog and emits grounding location tokens. We train on a real-world interactive dataset for IVG, and a mixed ambiguity set for the detector. With InViG-only supervision, our model surpasses a state-of-the-art method while using parameter-efficient fine-tuning. Similarly, the ambiguity detector outperforms prior baselines. Overall, CLUE turns the internal cross-modal attention of a VLM into an explicit, spatially grounded signal for deciding when to ask. The data and code are publicly available at: mouadabrini.github.io/clue