WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI

TL;DR

This paper introduces WeAudit, a system designed to scaffold end users and AI practitioners in AI auditing, enhancing user engagement and actionable insights through a structured workflow and user study.

Contribution

WeDeveloped WeAudit, a novel workflow and system that supports end users in AI auditing, addressing the challenge of effectively engaging users for actionable insights.

Findings

WeAudit helps users notice and reflect on AI harms.

Users can articulate findings that practitioners can act upon.

The system fosters collaborative AI auditing between users and practitioners.

Abstract

There has been growing interest from both practitioners and researchers in engaging end users in AI auditing, to draw upon users' unique knowledge and lived experiences. However, we know little about how to effectively scaffold end users in auditing in ways that can generate actionable insights for AI practitioners. Through formative studies with both users and AI practitioners, we first identified a set of design goals to support user-engaged AI auditing. We then developed WeAudit, a workflow and system that supports end users in auditing AI both individually and collectively. We evaluated WeAudit through a three-week user study with user auditors and interviews with industry Generative AI practitioners. Our findings offer insights into how WeAudit supports users in noticing and reflecting upon potential AI harms and in articulating their findings in ways that industry practitioners can act upon. Based on our observations and feedback from both users and practitioners, we identify several opportunities to better support user engagement in AI auditing processes. We discuss implications for future research to support effective and responsible user engagement in AI auditing and red-teaming.