Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems

TL;DR

This paper evaluates how uncertainty and fairness impact the reliability of LLM-based recommendation systems, introducing new benchmarks and datasets to improve fairness and interpretability.

Contribution

It introduces a comprehensive benchmark and dataset for fairness and uncertainty in LLM recommendations, along with novel evaluation methodologies and insights into bias patterns.

Findings

DeepMind's Gemini 1.5 Flash shows systematic unfairness for sensitive attributes.

Disparities persist under prompt perturbations like typos and multilingual inputs.

Personality-aware fairness reveals bias patterns and trade-offs with personalization.

Abstract

Large language models (LLMs) enable powerful zero-shot recommendations by leveraging broad contextual knowledge, yet predictive uncertainty and embedded biases threaten reliability and fairness. This paper studies how uncertainty and fairness evaluations affect the accuracy, consistency, and trustworthiness of LLM-generated recommendations. We introduce a benchmark of curated metrics and a dataset annotated for eight demographic attributes (31 categorical values) across two domains: movies and music. Through in-depth case studies, we quantify predictive uncertainty (via entropy) and demonstrate that Google DeepMind's Gemini 1.5 Flash exhibits systematic unfairness for certain sensitive attributes; measured similarity-based gaps are SNSR at 0.1363 and SNSV at 0.0507. These disparities persist under prompt perturbations such as typographical errors and multilingual inputs. We further integrate personality-aware fairness into the RecLLM evaluation pipeline to reveal personality-linked bias patterns and expose trade-offs between personalization and group fairness. We propose a novel uncertainty-aware evaluation methodology for RecLLMs, present empirical insights from deep uncertainty case studies, and introduce a personality profile-informed fairness benchmark that advances explainability and equity in LLM recommendations. Together, these contributions establish a foundation for safer, more interpretable RecLLMs and motivate future work on multi-model benchmarks and adaptive calibration for trustworthy deployment.