Human and LLM-Based Voice Assistant Interaction: An Analytical Framework for User Verbal and Nonverbal Behaviors

TL;DR

This paper develops an analytical framework to study verbal and nonverbal user behaviors during complex interactions with LLM-based voice assistants, addressing a gap in understanding human-VA communication dynamics.

Contribution

It introduces a novel three-dimensional analytical framework for examining verbal and nonverbal behaviors across interaction stages in human-LLM-VA interactions.

Findings

Identified key verbal and nonverbal behaviors in user interactions.

Mapped behavior transitions across exploration, conflict, and integration stages.

Provided a foundation for optimizing human-LLM-VA communication.

Abstract

Recent progress in large language model (LLM) technology has significantly enhanced the interaction experience between humans and voice assistants (VAs). This project aims to explore a user's continuous interaction with LLM-based VA (LLM-VA) during a complex task. We recruited 12 participants to interact with an LLM-VA during a cooking task, selected for its complexity and the requirement for continuous interaction. We observed that users show both verbal and nonverbal behaviors, though they know that the LLM-VA can not capture those nonverbal signals. Despite the prevalence of nonverbal behavior in human-human communication, there is no established analytical methodology or framework for exploring it in human-VA interactions. After analyzing 3 hours and 39 minutes of video recordings, we developed an analytical framework with three dimensions: 1) behavior characteristics, including both verbal and nonverbal behaviors, 2) interaction stages--exploration, conflict, and integration--that illustrate the progression of user interactions, and 3) stage transition throughout the task. This analytical framework identifies key verbal and nonverbal behaviors that provide a foundation for future research and practical applications in optimizing human and LLM-VA interactions.