Think-Aloud Verbalizations for Identifying User Experience Problems: Effects of Language Proficiency with Chinese Non-Native English Speakers

TL;DR

This study investigates whether Chinese non-native English speakers' think-aloud verbalizations reveal UX problems similarly to native speakers, finding comparable patterns and implications for AI-assisted UX analysis tools.

Contribution

It provides the first comparative analysis of native and non-native English speakers' think-aloud verbalizations in UX testing, informing AI tool design.

Findings

Similar verbalization patterns between native and non-native speakers

Language proficiency does not significantly alter verbalization-problem correlations

TA protocols have minimal impact on verbalization patterns

Abstract

Subtle patterns in users' think-aloud (TA) verbalizations (i.e., utterances) are shown to be telltale signs of user experience (UX) problems and used to build artificial intelligence (AI) models or AI-assisted tools to help UX evaluators identify UX problems automatically or semi-automatically. Despite the potential of such verbalization patterns, they were uncovered with native English speakers. As most people who speak English are non-native speakers, it is important to investigate whether similar patterns exist in non-native English speakers' TA verbalizations. As a first step to answer this question, we conducted think-aloud usability testing with Chinese non-native English speakers and native English speakers using three common TA protocols. We compared their verbalizations and UX problems that they encountered to understand the effects of language and TA protocols. Our findings show that both language groups had similar amounts and proportions of verbalization categories, encountered similar problems, and had similar verbalization patterns that indicate UX problems. Furthermore, TA protocols did not significantly affect the correlations between verbalizations and problems. Based on the findings, we present three design implications for UX practitioners and the design of AI-assisted analysis tools.