Measuring Voice UX Quantitatively: A Rapid Review

TL;DR

This paper reviews existing methods for quantitatively measuring voice user experience, highlighting inconsistencies and proposing frameworks to standardize and improve future research in this emerging field.

Contribution

It provides a comprehensive review of current voice UX measurement approaches and introduces two frameworks to guide future standardization efforts.

Findings

Little consensus on measurement methods

Overreliance on lab studies and unvalidated scales

Proposes frameworks for standardized voice UX measurement

Abstract

Computer voice is experiencing a renaissance through the growing popularity of voice-based interfaces, agents, and environments. Yet, how to measure the user experience (UX) of voice-based systems remains an open and urgent question, especially given that their form factors and interaction styles tend to be non-visual, intangible, and often considered disembodied or "body-less." As a first step, we surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed important for voice UX. Our findings show that there is little consensus, even with similar situations and systems, as well as an overreliance on lab work and unvalidated scales. In response, we offer two high-level descriptive frameworks for guiding future research, developing standardized instruments, and informing ongoing review work. Our work highlights the current strengths and weaknesses of voice UX research and charts a path towards measuring voice UX in a more comprehensive way.