Pardon? Evaluating Conversational Repair in Large Audio-Language Models

TL;DR

This paper introduces a new evaluation framework for Large Audio-Language Models that assesses their ability to recognize unanswerable inputs and perform conversational repairs, highlighting limitations of current accuracy-focused metrics.

Contribution

It proposes the EAR score, a novel metric for evaluating both answerability and repair behavior, and demonstrates its effectiveness through experiments on spoken QA benchmarks.

Findings

Models perform well on answerable inputs but struggle with unanswerability detection.

Current metrics overlook the importance of conversational repair in real-world interactions.

The study reveals a gap between answer accuracy and conversational reliability in LALMs.

Abstract

Large Audio-Language Models (LALMs) have demonstrated strong performance in spoken question answering (QA), with existing evaluations primarily focusing on answer accuracy and robustness to acoustic perturbations. However, such evaluations implicitly assume that spoken inputs remain semantically answerable, an assumption that often fails in real-world interaction when essential information is missing. In this work, we introduce a repair-aware evaluation setting that explicitly distinguishes between answerable and unanswerable audio inputs. We define answerability as a property of the input itself and construct paired evaluation conditions using a semantic-acoustic masking protocol. Based on this setting, we propose the Evaluability Awareness and Repair (EAR) score, a non-compensatory metric that jointly evaluates task competence under answerable conditions and repair behavior under unanswerable conditions. Experiments on two spoken QA benchmarks across diverse LALMs reveal a consistent gap between answer accuracy and conversational reliability: while many models perform well when inputs are answerable, most fail to recognize semantic unanswerability and initiate appropriate conversational repair. These findings expose a limitation of prevailing accuracy-centric evaluation practices and motivate reliability assessments that treat unanswerable inputs as cues for repair and continued interaction.