# Revealing Potential Biases in LLM-Based Recommender Systems in the Cold Start Setting

## TL;DR

This paper introduces a benchmark to evaluate fairness in LLM-based recommender systems in cold-start scenarios, revealing consistent societal biases and complex relationships between model size and fairness.

## Contribution

It presents a modular, configurable benchmark for systematic fairness evaluation of open-source LLM recommenders in cold-start settings.

## Key findings

- State-of-the-art models exhibit gendered and cultural biases.
- Biases are consistent across different recommendation domains.
- Model size has a non-linear impact on fairness.

## Abstract

Large Language Models (LLMs) are increasingly used for recommendation tasks due to their general-purpose capabilities. While LLMs perform well in rich-context settings, their behavior in cold-start scenarios, where only limited signals such as age, gender, or language are available, raises fairness concerns because they may rely on societal biases encoded during pretraining. We introduce a benchmark specifically designed to evaluate fairness in zero-context recommendation. Our modular pipeline supports configurable recommendation domains and sensitive attributes, enabling systematic and flexible audits of any open-source LLM. Through evaluations of state-of-the-art models (Gemma 3 and Llama 3.2), we uncover consistent biases across recommendation domains (music, movies, and colleges) including gendered and cultural stereotypes. We also reveal a non-linear relationship between model size and fairness, highlighting the need for nuanced analysis.