Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody

TL;DR

This paper introduces a novel speech prosody-based method for improving robot understanding of spoken instructions, achieving high accuracy in intent detection and disambiguation, and provides a new dataset for research in speech disambiguation in robotics.

Contribution

It presents a new approach leveraging speech prosody for intent inference, integrating it with large language models, and introduces the first ambiguous speech dataset for robotics.

Findings

95.79% accuracy in referent intent detection

71.96% accuracy in task plan disambiguation

First dataset for speech disambiguation in robotics

Abstract

Enabling robots to accurately interpret and execute spoken language instructions is essential for effective human-robot collaboration. Traditional methods rely on speech recognition to transcribe speech into text, often discarding crucial prosodic cues needed for disambiguating intent. We propose a novel approach that directly leverages speech prosody to infer and resolve instruction intent. Predicted intents are integrated into large language models via in-context learning to disambiguate and select appropriate task plans. Additionally, we present the first ambiguous speech dataset for robotics, designed to advance research in speech disambiguation. Our method achieves 95.79% accuracy in detecting referent intents within an utterance and determines the intended task plan of ambiguous instructions with 71.96% accuracy, demonstrating its potential to significantly improve human-robot communication.