No that's not what I meant: Handling Third Position Repair in Conversational Question Answering

TL;DR

This paper introduces Repair-QA, a large dataset of Third Position Repairs in conversational QA, and evaluates models' ability to handle miscommunication correction, highlighting challenges and improvements with LLMs.

Contribution

The paper presents the first large dataset of TPRs in conversational QA and benchmarks models' performance in executing and understanding TPRs.

Findings

GPT-3 models perform poorly on TPRs out-of-the-box.

Fine-tuning on Repair-QA significantly improves GPT-3's TPR handling.

Models show potential to better manage miscommunication in dialogue.

Abstract

The ability to handle miscommunication is crucial to robust and faithful conversational AI. People usually deal with miscommunication immediately as they detect it, using highly systematic interactional mechanisms called repair. One important type of repair is Third Position Repair (TPR) whereby a speaker is initially misunderstood but then corrects the misunderstanding as it becomes apparent after the addressee's erroneous response. Here, we collect and publicly release Repair-QA, the first large dataset of TPRs in a conversational question answering (QA) setting. The data is comprised of the TPR turns, corresponding dialogue contexts, and candidate repairs of the original turn for execution of TPRs. We demonstrate the usefulness of the data by training and evaluating strong baseline models for executing TPRs. For stand-alone TPR execution, we perform both automatic and human evaluations on a fine-tuned T5 model, as well as OpenAI's GPT-3 LLMs. Additionally, we extrinsically evaluate the LLMs' TPR processing capabilities in the downstream conversational QA task. The results indicate poor out-of-the-box performance on TPR's by the GPT-3 models, which then significantly improves when exposed to Repair-QA.