The PUEVA Inventory: A Toolkit to Evaluate the Personality, Usability and Enjoyability of Voice Agents

TL;DR

The paper introduces the PUEVA toolkit, a comprehensive questionnaire designed to evaluate personality, usability, and enjoyability in voice user interfaces, facilitating subjective experience assessment beyond technical performance.

Contribution

It presents a novel, validated toolkit for measuring subjective enjoyment and personality traits in voice agents, based on extensive literature review and empirical studies.

Findings

The PUEVA questionnaire reliably measures personality, usability, and enjoyability.

It enables comparison of user experiences across different voice agents.

The toolkit supports both between-subject and within-subject research designs.

Abstract

The proliferation of voice agents in consumer devices requires new tools for evaluating these systems beyond their technical functionality. This paper presents a toolkit for the evaluation of Voice User Interfaces (VUIs) with the intention of measuring the crucial factors of subjective enjoyment in the user experience. The PUEVA toolkit was constructed using a meta-analysis of existing literature, structured N=20 and semi-structured N=18 interviews and a within subjects lab study. The resulting questionnaire contains 35 items that represent 12 scales in three categories: (1) Personality (2) Usablity and (3) Enjoyability. The PUEVA Toolkit moves us towards the capacity to evaluate and compare subjective, joyful experiences in between-subject as well as within-subject research designs.