Beyond Words: Measuring User Experience through Speech Analysis in Voice User Interfaces

TL;DR

This study demonstrates that speech analysis can serve as a real-time, implicit measure of user experience in voice interfaces, correlating speech features with satisfaction and enabling adaptive interactions.

Contribution

The paper introduces a novel approach to assessing user experience through speech analysis, providing evidence that speech features can predict UX levels and support adaptive voice UI systems.

Findings

Speech features correlate with self-reported satisfaction.

Machine learning models classify UX levels with promising accuracy.

Speech analysis offers a real-time, implicit UX measurement method.

Abstract

Voice assistants (VAs) are typically evaluated through task performance metrics and self-report questionnaires, but people's voices themselves carry rich paralinguistic cues that reveal affect, effort, and interaction breakdowns. We present a within-subjects study (N=49) that systematically compared three VA personas across three usage scenarios to investigate whether speech-derived audio features can serve as a proxy for user experience (UX). Participants' speech was analyzed for temporal, spectral, and linguistic markers, alongside standardized UX measures, brief mood and stress ratings, and a post-study questionnaire. We found correlations between specific speech features and self-reported satisfaction and experience. Furthermore, a machine learning model trained on speech features achieved promising accuracy in classifying UX levels, indicating that this might be a reasonable alternative to self-report instruments. Our findings establish speech as a viable, real-time signal for implicitly measuring UX and point toward adaptive VUIs that respond dynamically to emotional and usability-related vocal cues.