QueerGen: How LLMs Reflect Societal Norms on Gender and Sexuality in Sentence Completion Tasks

TL;DR

This study investigates how Large Language Models reproduce societal norms and biases related to gender and sexuality, revealing that model type and access influence the nature and extent of these biases in text generation.

Contribution

It provides a systematic analysis of societal norm reproduction in LLMs, highlighting how different models exhibit biases and the impact of model characteristics on representational harms.

Findings

MLMs show more negative sentiment and toxicity towards queer-marked subjects

ARLMs partially reduce biases but still reproduce societal norms

Closed-access ARLMs may produce more harmful outputs for unmarked subjects

Abstract

This paper examines how Large Language Models (LLMs) reproduce societal norms, particularly heterocisnormativity, and how these norms translate into measurable biases in their text generations. We investigate whether explicit information about a subject's gender or sexuality influences LLM responses across three subject categories: queer-marked, non-queer-marked, and the normalized "unmarked" category. Representational imbalances are operationalized as measurable differences in English sentence completions across four dimensions: sentiment, regard, toxicity, and prediction diversity. Our findings show that Masked Language Models (MLMs) produce the least favorable sentiment, higher toxicity, and more negative regard for queer-marked subjects. Autoregressive Language Models (ARLMs) partially mitigate these patterns, while closed-access ARLMs tend to produce more harmful outputs for unmarked subjects. Results suggest that LLMs reproduce normative social assumptions, though the form and degree of bias depend strongly on specific model characteristics, which may redistribute, but not eliminate, representational harms.