Should AI Mimic People? Understanding AI-Supported Writing Technology Among Black Users

TL;DR

This study explores how Black American users perceive AI-supported writing tools, revealing a tension between their benefits and feelings of alienation due to cultural biases in the technology.

Contribution

It provides qualitative insights into Black users' perceptions of AISWT, highlighting cultural biases and the emotional impact of AI suggestions on marginalized communities.

Findings

Users feel AI tools often fail to recognize African American Vernacular English.

AI suggestions can be perceived as hurtful and alienating.

There is a tension between AI's cultural bias and efforts to mimic Black language.

Abstract

AI-supported writing technologies (AISWT) that provide grammatical suggestions, autocomplete sentences, or generate and rewrite text are now a regular feature integrated into many people's workflows. However, little is known about how people perceive the suggestions these tools provide. In this paper, we investigate how Black American users perceive AISWT, motivated by prior findings in natural language processing that highlight how the underlying large language models can contain racial biases. Using interviews and observational user studies with 13 Black American users of AISWT, we found a strong tradeoff between the perceived benefits of using AISWT to enhance their writing style and feeling like "it wasn't built for us". Specifically, participants reported AISWT's failure to recognize commonly used names and expressions in African American Vernacular English, experiencing its corrections as hurtful and alienating and fearing it might further minoritize their culture. We end with a reflection on the tension between AISWT that fail to include Black American culture and language, and AISWT that attempt to mimic it, with attention to accuracy, authenticity, and the production of social difference.