Stereotype or Personalization? User Identity Biases Chatbot Recommendations

TL;DR

This paper investigates how large language models generate recommendations influenced by user identity, revealing racial stereotypes and lack of transparency, and emphasizes the need for clearer disclosure of identity-based biases in chatbot responses.

Contribution

It demonstrates that LLMs produce racially stereotypical recommendations influenced by user identity cues and highlights the lack of transparency in these biases.

Findings

User identity significantly influences recommendations (p < 0.001)

Models produce racially stereotypical responses regardless of explicit or implicit cues

Current chatbots fail to disclose when recommendations are influenced by user identity

Abstract

While personalized recommendations are often desired by users, it can be difficult in practice to distinguish cases of bias from cases of personalization: we find that models generate racially stereotypical recommendations regardless of whether the user revealed their identity intentionally through explicit indications or unintentionally through implicit cues. We demonstrate that when people use large language models (LLMs) to generate recommendations, the LLMs produce responses that reflect both what the user wants and who the user is. We argue that chatbots ought to transparently indicate when recommendations are influenced by a user's revealed identity characteristics, but observe that they currently fail to do so. Our experiments show that even though a user's revealed identity significantly influences model recommendations (p < 0.001), model responses obfuscate this fact in response to user queries. This bias and lack of transparency occurs consistently across multiple popular consumer LLMs and for four American racial groups.