"It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study

TL;DR

This study explores how generative AI tools influence data analysis workflows, highlighting benefits in information gathering and sensemaking, while also identifying challenges like query formulation and result verification through participatory prompting with Bing Chat.

Contribution

It demonstrates the effectiveness of participatory prompting in studying AI-assisted data analysis and reveals specific ways AI can both aid and hinder the process.

Findings

AI improves information foraging and sensemaking in data analysis.

Challenges include difficulties in query formulation and verifying AI-generated results.

Participatory prompting is a valuable method for studying AI-human collaboration.

Abstract

Generative AI tools can help users with many tasks. One such task is data analysis, which is notoriously challenging for non-expert end-users due to its expertise requirements, and where AI holds much potential, such as finding relevant data sources, proposing analysis strategies, and writing analysis code. To understand how data analysis workflows can be assisted or impaired by generative AI, we conducted a study (n=15) using Bing Chat via participatory prompting. Participatory prompting is a recently developed methodology in which users and researchers reflect together on tasks through co-engagement with generative AI. In this paper we demonstrate the value of the participatory prompting method. We found that generative AI benefits the information foraging and sensemaking loops of data analysis in specific ways, but also introduces its own barriers and challenges, arising from the difficulties of query formulation, specifying context, and verifying results.