Learning AI Auditing: A Case Study of Teenagers Auditing a Generative AI Model

TL;DR

This paper explores how teenagers can participate in AI auditing through a participatory workshop, revealing their engagement, creativity, and ability to identify biases in a popular generative AI tool, thereby fostering AI literacy.

Contribution

It demonstrates that teenagers can effectively conduct algorithm audits, uncover biases, and contribute new insights, expanding the scope of AI literacy and participatory auditing methods.

Findings

Teenagers identified biases in age, race, and gender representation.

Participants engaged creatively and independently in the auditing process.

Their findings were comparable to expert analyses, supporting participatory auditing.

Abstract

This study investigates how high school-aged youth engage in algorithm auditing to identify and understand biases in artificial intelligence and machine learning (AI/ML) tools they encounter daily. With AI/ML technologies being increasingly integrated into young people's lives, there is an urgent need to equip teenagers with AI literacies that build both technical knowledge and awareness of social impacts. Algorithm audits (also called AI audits) have traditionally been employed by experts to assess potential harmful biases, but recent research suggests that non-expert users can also participate productively in auditing. We conducted a two-week participatory design workshop with 14 teenagers (ages 14-15), where they audited the generative AI model behind TikTok's Effect House, a tool for creating interactive TikTok filters. We present a case study describing how teenagers approached the audit, from deciding what to audit to analyzing data using diverse strategies and communicating their results. Our findings show that participants were engaged and creative throughout the activities, independently raising and exploring new considerations, such as age-related biases, that are uncommon in professional audits. We drew on our expertise in algorithm auditing to triangulate their findings as a way to examine if the workshop supported participants to reach coherent conclusions in their audit. Although the resulting number of changes in race, gender, and age representation uncovered by the teens were slightly different from ours, we reached similar conclusions. This study highlights the potential for auditing to inspire learning activities to foster AI literacies, empower teenagers to critically examine AI systems, and contribute fresh perspectives to the study of algorithmic harms.