Human Latency Conversational Turns for Spoken Avatar Systems

TL;DR

This paper explores methods for enabling spoken avatar systems to generate responses in near real-time by predicting missing parts of utterances, aiming to match human conversational latencies.

Contribution

It introduces techniques for understanding incomplete utterances and generating responses quickly, utilizing GPT-4 to fill in missing context and proposing a classifier to detect semantic completeness.

Findings

GPT-4 can fill in missing context over 60% of the time

A simple classifier can determine if an utterance is complete or needs filler

Methods enable near real-time responses matching human dialogue timing

Abstract

A problem with many current Large Language Model (LLM) driven spoken dialogues is the response time. Some efforts such as Groq address this issue by lightning fast processing of the LLM, but we know from the cognitive psychology literature that in human-to-human dialogue often responses occur prior to the speaker completing their utterance. No amount of delay for LLM processing is acceptable if we wish to maintain human dialogue latencies. In this paper, we discuss methods for understanding an utterance in close to real time and generating a response so that the system can comply with human-level conversational turn delays. This means that the information content of the final part of the speaker's utterance is lost to the LLM. Using the Google NaturalQuestions (NQ) database, our results show GPT-4 can effectively fill in missing context from a dropped word at the end of a question over 60% of the time. We also provide some examples of utterances and the impacts of this information loss on the quality of LLM response in the context of an avatar that is currently under development. These results indicate that a simple classifier could be used to determine whether a question is semantically complete, or requires a filler phrase to allow a response to be generated within human dialogue time constraints.