Pointing out Human Answer Mistakes in a Goal-Oriented Visual Dialogue

TL;DR

This paper investigates how human answer mistakes in visual dialogue depend on question type and turn, and demonstrates methods for agents to point out these mistakes to improve interaction accuracy.

Contribution

It introduces an analysis of human mistakes in visual dialogue and evaluates models for pointing out errors, reflecting more realistic human-agent communication scenarios.

Findings

Human mistakes vary with question type and dialogue turn.

Model accuracy improves when pointing out human mistakes.

Simple models can effectively identify and highlight human errors.

Abstract

Effective communication between humans and intelligent agents has promising applications for solving complex problems. One such approach is visual dialogue, which leverages multimodal context to assist humans. However, real-world scenarios occasionally involve human mistakes, which can cause intelligent agents to fail. While most prior research assumes perfect answers from human interlocutors, we focus on a setting where the agent points out unintentional mistakes for the interlocutor to review, better reflecting real-world situations. In this paper, we show that human answer mistakes depend on question type and QA turn in the visual dialogue by analyzing a previously unused data collection of human mistakes. We demonstrate the effectiveness of those factors for the model's accuracy in a pointing-human-mistake task through experiments using a simple MLP model and a Visual Language Model.