Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models

TL;DR

This paper introduces BlockWorld-Repairs, a new dataset for evaluating vision and language models' ability to handle user corrections in multi-modal tasks, revealing current models' limitations and proposing fine-tuning improvements.

Contribution

The paper presents a novel dataset for multi-modal repair sequences and evaluates state-of-the-art models, highlighting their deficiencies and proposing targeted fine-tuning methods to improve performance.

Findings

Models underperform compared to humans in repair tasks.

Specialised loss functions improve model performance.

Models still struggle with generalising repairs in new scenarios.

Abstract

In dialogue, the addressee may initially misunderstand the speaker and respond erroneously, often prompting the speaker to correct the misunderstanding in the next turn with a Third Position Repair (TPR). The ability to process and respond appropriately to such repair sequences is thus crucial in conversational AI systems. In this paper, we first collect, analyse, and publicly release BlockWorld-Repairs: a dataset of multi-modal TPR sequences in an instruction-following manipulation task that is, by design, rife with referential ambiguity. We employ this dataset to evaluate several state-of-the-art Vision and Language Models (VLM) across multiple settings, focusing on their capability to process and accurately respond to TPRs and thus recover from miscommunication. We find that, compared to humans, all models significantly underperform in this task. We then show that VLMs can benefit from specialised losses targeting relevant tokens during fine-tuning, achieving better performance and generalising better to new scenarios. Our results suggest that these models are not yet ready to be deployed in multi-modal collaborative settings where repairs are common, and highlight the need to design training regimes and objectives that facilitate learning from interaction. Our code and data are available at www.github.com/JChiyah/blockworld-repairs