A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System

TL;DR

This paper introduces a normative framework to evaluate consumer fairness in large language model recommender systems, addressing biases and fairness challenges that traditional methods overlook, with experimental validation on the MovieLens dataset.

Contribution

It proposes a formal, structured approach to benchmark fairness in RecLLMs, filling a gap left by classical fairness evaluations in traditional recommender systems.

Findings

Fairness deviations observed in age-based recommendations under different in-context learning settings

Statistical tests confirm these deviations are significant and not due to chance

Highlights the importance of robust fairness evaluation methods for RecLLMs

Abstract

The rapid adoption of large language models (LLMs) in recommender systems (RS) presents new challenges in understanding and evaluating their biases, which can result in unfairness or the amplification of stereotypes. Traditional fairness evaluations in RS primarily focus on collaborative filtering (CF) settings, which may not fully capture the complexities of LLMs, as these models often inherit biases from large, unregulated data. This paper proposes a normative framework to benchmark consumer fairness in LLM-powered recommender systems (RecLLMs). We critically examine how fairness norms in classical RS fall short in addressing the challenges posed by LLMs. We argue that this gap can lead to arbitrary conclusions about fairness, and we propose a more structured, formal approach to evaluate fairness in such systems. Our experiments on the MovieLens dataset on consumer fairness, using in-context learning (zero-shot vs. few-shot) reveal fairness deviations in age-based recommendations, particularly when additional contextual examples are introduced (ICL-2). Statistical significance tests confirm that these deviations are not random, highlighting the need for robust evaluation methods. While this work offers a preliminary discussion on a proposed normative framework, our hope is that it could provide a formal, principled approach for auditing and mitigating bias in RecLLMs. The code and dataset used for this work will be shared at "gihub-anonymized".