Facilitating Longitudinal Interaction Studies of AI Systems

TL;DR

This paper discusses the importance and challenges of conducting longitudinal interaction studies of AI systems in UIST research, proposing strategies and community efforts to facilitate such long-term evaluations.

Contribution

It introduces practical approaches and community initiatives to support longitudinal studies of AI systems in UIST research, addressing deployment and evaluation challenges.

Findings

Workshop successfully engaged researchers in longitudinal study design

Practical strategies were developed for long-term data collection

Community building efforts promoted adoption of longitudinal methods

Abstract

UIST researchers develop tools to address user challenges. However, user interactions with AI evolve over time through learning, adaptation, and repurposing, making one time evaluations insufficient. Capturing these dynamics requires longer-term studies, but challenges in deployment, evaluation design, and data collection have made such longitudinal research difficult to implement. Our workshop aims to tackle these challenges and prepare researchers with practical strategies for longitudinal studies. The workshop includes a keynote, panel discussions, and interactive breakout groups for discussion and hands-on protocol design and tool prototyping sessions. We seek to foster a community around longitudinal system research and promote it as a more embraced method for designing, building, and evaluating UIST tools.