'What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges

TL;DR

This paper evaluates how multi-modal dialogue models handle clarificational exchanges, revealing that language models excel with dialogue history while multi-modal models benefit from disentangled object representations for complex ambiguities.

Contribution

It introduces a framework to assess multi-modal dialogue models' ability to process clarificational exchanges and highlights the importance of disentangled object representations.

Findings

Language models encode simple multi-modal semantics.

Multi-modal models improve with disentangled object representations.

Models differ in handling complex referential ambiguities.

Abstract

Referential ambiguities arise in dialogue when a referring expression does not uniquely identify the intended referent for the addressee. Addressees usually detect such ambiguities immediately and work with the speaker to repair it using meta-communicative, Clarificational Exchanges (CE): a Clarification Request (CR) and a response. Here, we argue that the ability to generate and respond to CRs imposes specific constraints on the architecture and objective functions of multi-modal, visually grounded dialogue models. We use the SIMMC 2.0 dataset to evaluate the ability of different state-of-the-art model architectures to process CEs, with a metric that probes the contextual updates that arise from them in the model. We find that language-based models are able to encode simple multi-modal semantic information and process some CEs, excelling with those related to the dialogue history, whilst multi-modal models can use additional learning objectives to obtain disentangled object representations, which become crucial to handle complex referential ambiguities across modalities overall.