My LLM might Mimic AAE -- But When Should it?

TL;DR

This study investigates Black Americans' perceptions of LLMs' ability to authentically produce African American English (AAE) and their preferences for when such language use is appropriate, emphasizing the importance of context and user autonomy.

Contribution

It provides empirical insights into Black Americans' preferences and perceptions regarding LLM-generated AAE, highlighting the importance of context and user control in language authenticity.

Findings

Black Americans prefer LLMs to default to Mainstream U.S. English in formal settings.

Participants find LLM-produced AAE authentic when properly prompted and contextualized.

Preference for AAE use varies with setting and user choice.

Abstract

We examine the representation of African American English (AAE) in large language models (LLMs), exploring (a) the perceptions Black Americans have of how effective these technologies are at producing authentic AAE, and (b) in what contexts Black Americans find this desirable. Through both a survey of Black Americans ($n=$ 104) and annotation of LLM-produced AAE by Black Americans ($n=$ 228), we find that Black Americans favor choice and autonomy in determining when AAE is appropriate in LLM output. They tend to prefer that LLMs default to communicating in Mainstream U.S. English in formal settings, with greater interest in AAE production in less formal settings. When LLMs were appropriately prompted and provided in context examples, our participants found their outputs to have a level of AAE authenticity on par with transcripts of Black American speech. Select code and data for our project can be found here: https://github.com/smelliecat/AAEMime.git