Probing Cultural Signals in Large Language Models through Author Profiling

TL;DR

This paper investigates cultural biases in large language models by evaluating their ability to perform author profiling on song lyrics, revealing systematic ethnic biases and proposing fairness metrics.

Contribution

It introduces a novel zero-shot author profiling method for LLMs on song lyrics and quantifies cultural biases using new fairness metrics.

Findings

LLMs achieve non-trivial profiling accuracy

Models show systematic North American bias

DeepSeek-1.5B aligns more with Asian ethnicity

Abstract

Large language models (LLMs) are increasingly deployed in applications with societal impact, raising concerns about the cultural biases they encode. We probe these representations by evaluating whether LLMs can perform author profiling from song lyrics in a zero-shot setting, inferring singers' gender and ethnicity without task-specific fine-tuning. Across several open-source models evaluated on more than 10,000 lyrics, we find that LLMs achieve non-trivial profiling performance but demonstrate systematic cultural alignment: most models default toward North American ethnicity, while DeepSeek-1.5B aligns more strongly with Asian ethnicity. This finding emerges from both the models' prediction distributions and an analysis of their generated rationales. To quantify these disparities, we introduce two fairness metrics, Modality Accuracy Divergence (MAD) and Recall Divergence (RD), and show that Ministral-8B displays the strongest ethnicity bias among the evaluated models, whereas Gemma-12B shows the most balanced behavior. Our code is available on [GitHub](https://github.com/ValentinLafargue/CulturalProbingLLM) and results on [HuggingFace](https://huggingface.co/datasets/ValentinLAFARGUE/AuthorProfilingResults).