Provocation: Who benefits from "inclusion" in Generative AI?

TL;DR

This paper critically examines who truly benefits from inclusion efforts in generative AI, highlighting systemic barriers and the need for explicit evaluation of benefits and harms for marginalized groups.

Contribution

It introduces a speculative case study to identify barriers and emphasizes the importance of explicit benefit-harm analysis in participatory AI evaluation.

Findings

Dominant participation structures may overlook marginalized groups' experiences.

Explicit benefit-harm analysis is crucial for equitable AI development.

Systemic change is necessary to realize true inclusion benefits.

Abstract

The demands for accurate and representative generative AI systems means there is an increased demand on participatory evaluation structures. While these participatory structures are paramount to to ensure non-dominant values, knowledge and material culture are also reflected in AI models and the media they generate, we argue that dominant structures of community participation in AI development and evaluation are not explicit enough about the benefits and harms that members of socially marginalized groups may experience as a result of their participation. Without explicit interrogation of these benefits by AI developers, as a community we may remain blind to the immensity of systemic change that is needed as well. To support this provocation, we present a speculative case study, developed from our own collective experiences as AI researchers. We use this speculative context to itemize the barriers that need to be overcome in order for the proposed benefits to marginalized communities to be realized, and harms mitigated.