The Effects of Interactive AI Design on User Behavior: An Eye-tracking Study of Fact-checking COVID-19 Claims

TL;DR

This study investigates how interactive AI design influences user behavior in fact-checking COVID-19 claims, revealing that interactivity affects attention and dwell time but not mental workload, highlighting its positive role.

Contribution

It provides empirical evidence on the impact of AI interactivity on user attention and engagement during fact-checking tasks.

Findings

Interactivity affects dwell time and eye fixations on key areas.

Participants spent more time evaluating claims with interactive AI.

Mental workload was unaffected by interactivity.

Abstract

We conducted a lab-based eye-tracking study to investigate how the interactivity of an AI-powered fact-checking system affects user interactions, such as dwell time, attention, and mental resources involved in using the system. A within-subject experiment was conducted, where participants used an interactive and a non-interactive version of a mock AI fact-checking system and rated their perceived correctness of COVID-19 related claims. We collected web-page interactions, eye-tracking data, and mental workload using NASA-TLX. We found that the presence of the affordance of interactively manipulating the AI system's prediction parameters affected users' dwell times, and eye-fixations on AOIs, but not mental workload. In the interactive system, participants spent the most time evaluating claims' correctness, followed by reading news. This promising result shows a positive role of interactivity in a mixed-initiative AI-powered system.