"It Became My Buddy, But I'm Not Afraid to Disagree": A Multi-Session Study of UX Evaluators Collaborating with Conversational AI Assistants

TL;DR

This study investigates how UX evaluators interact with conversational AI assistants of varying perceived expertise over multiple sessions, revealing trust dynamics, behavioral shifts, and implications for AI design in usability testing.

Contribution

It provides empirical insights into the evolving perceptions and strategies of UX evaluators working with conversational AI of different expertise levels across multiple sessions.

Findings

Trust dips and recovers over sessions

Efficiency improves with experience

Evaluators favor experienced AI despite initial perceptions

Abstract

AI-assisted usability analysis can potentially reduce the time and effort of finding usability problems, yet little is known about how AI's perceived expertise influences evaluators' analytic strategies and perceptions over time. We ran a within-subjects, five-session study (six hours per participant) with 12 professional UX evaluators who worked with two conversational assistants designed to appear novice- or expert-like (differing in suggestion quantity and response accuracy). We logged behavioral measures (number of passes, suggestion acceptance rate), collected subjective ratings (trust, perceived efficiency), and conducted semi-structured interviews. Participants experienced an initial novelty effect and a subsequent dip in trust that recovered over time. Their efficiency improved as they shifted from a two-pass to a one-pass video inspection approach. Evaluators ultimately rated the experienced CA as significantly more efficient, trustworthy, and comprehensive, despite not perceiving expertise differences early on. We conclude with design implications for adapting AI expertise to enable calibrated human-AI collaboration.