What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages

TL;DR

This study uses multilingual LLMs to explore grammatical gender biases, revealing that models exhibit similar biases across languages despite different descriptive behaviors.

Contribution

It introduces a novel psycholinguistic approach to analyze grammatical gender bias in multilingual LLMs, demonstrating cross-language bias transferability.

Findings

A simple classifier predicts grammatical gender above chance.

LLMs show similar gender biases across different languages.

Models describe nouns differently but exhibit consistent biases.

Abstract

This paper investigates biases of Large Language Models (LLMs) through the lens of grammatical gender. Drawing inspiration from seminal works in psycholinguistics, particularly the study of gender's influence on language perception, we leverage multilingual LLMs to revisit and expand upon the foundational experiments of Boroditsky (2003). Employing LLMs as a novel method for examining psycholinguistic biases related to grammatical gender, we prompt a model to describe nouns with adjectives in various languages, focusing specifically on languages with grammatical gender. In particular, we look at adjective co-occurrences across gender and languages, and train a binary classifier to predict grammatical gender given adjectives an LLM uses to describe a noun. Surprisingly, we find that a simple classifier can not only predict noun gender above chance but also exhibit cross-language transferability. We show that while LLMs may describe words differently in different languages, they are biased similarly.