Beyond Text: Utilizing Vocal Cues to Improve Decision Making in LLMs for Robot Navigation Tasks

TL;DR

This paper introduces 'Beyond Text,' a method that enhances LLM decision-making in robot navigation by integrating vocal cues and audio features, leading to improved accuracy and robustness in human-robot interactions.

Contribution

The paper presents a novel approach that combines audio transcription with paralinguistic features to improve LLM performance in social navigation tasks.

Findings

Achieves a 70.26% winning rate, outperforming existing LLMs by up to 48.30%.

Enhances robustness against token manipulation attacks, with a 22.44% smaller decrease in success rate.

Advances human-robot interaction by integrating audio cues with text-based guidance.

Abstract

While LLMs excel in processing text in these human conversations, they struggle with the nuances of verbal instructions in scenarios like social navigation, where ambiguity and uncertainty can erode trust in robotic and other AI systems. We can address this shortcoming by moving beyond text and additionally focusing on the paralinguistic features of these audio responses. These features are the aspects of spoken communication that do not involve the literal wording (lexical content) but convey meaning and nuance through how something is said. We present Beyond Text: an approach that improves LLM decision-making by integrating audio transcription along with a subsection of these features, which focus on the affect and more relevant in human-robot conversations.This approach not only achieves a 70.26% winning rate, outperforming existing LLMs by 22.16% to 48.30% (gemini-1.5-pro and gpt-3.5 respectively), but also enhances robustness against token manipulation adversarial attacks, highlighted by a 22.44% less decrease ratio than the text-only language model in winning rate. Beyond Text' marks an advancement in social robot navigation and broader Human-Robot interactions, seamlessly integrating text-based guidance with human-audio-informed language models.