PointAloud: An Interaction Suite for AI-Supported Pointer-Centric Think-Aloud Computing

TL;DR

PointAloud introduces AI-driven, pointer-centric interactions to enhance Think-Aloud Computing by improving user engagement, providing low-distraction feedback, and supporting rich documentation, thereby facilitating better human-AI collaboration.

Contribution

The paper presents a novel suite of pointer-centric, AI-supported interactions that address practical challenges in Think-Aloud Computing, enabling more effective verbalization and documentation.

Findings

Participants found pointer-centric interactions valuable for documentation.

AI support increased user engagement and reduced distractions.

The approach facilitated richer work process insights.

Abstract

Think-Aloud Computing, a method for capturing users' verbalized thoughts during software tasks, allows eliciting rich contextual insights into evolving intentions, struggles, and decision-making processes of users in real-time. However, existing approaches face practical challenges: users often lack awareness of what is captured by the system, are not effectively encouraged to speak, and miss or are interrupted by system feedback. Additionally, thinking aloud should feel worthwhile for users due to the gained contextual AI assistance. To better support and harness Think-Aloud Computing, we introduce PointAloud, a suite of novel AI-driven pointer-centric interactions for in-the-moment verbalization encouragement, low-distraction system feedback, and contextually rich work process documentation alongside proactive AI assistance. Our user study with 12 participants provides insights into the value of pointer-centric think-aloud computing for work process documentation and human-AI co-creation. We conclude by discussing the broader implications of our findings and design considerations for pointer-centric and AI-supported Think-Aloud Computing workflows.