Biases in LLM-Generated Musical Taste Profiles for Recommendation

TL;DR

This paper investigates the accuracy and biases of LLM-generated musical taste profiles in recommendation systems, highlighting their potential for transparency and user control but also their limitations due to societal biases.

Contribution

It provides an empirical analysis of biases in LLM-generated profiles and their impact on user perception and recommendation quality in music streaming.

Findings

Profiles are biased by user attributes like taste diversity and mainstreamness.

Biases affect user identification with profiles and recommendation outcomes.

LLM profiles show potential but are limited by societal biases.

Abstract

One particularly promising use case of Large Language Models (LLMs) for recommendation is the automatic generation of Natural Language (NL) user taste profiles from consumption data. These profiles offer interpretable and editable alternatives to opaque collaborative filtering representations, enabling greater transparency and user control. However, it remains unclear whether users consider these profiles to be an accurate representation of their taste, which is crucial for trust and usability. Moreover, because LLMs inherit societal and data-driven biases, profile quality may systematically vary across user and item characteristics. In this paper, we study this issue in the context of music streaming, where personalization is challenged by a large and culturally diverse catalog. We conduct a user study in which participants rate NL profiles generated from their own listening histories. We analyze whether identification with the profiles is biased by user attributes (e.g., mainstreamness, taste diversity) and item features (e.g., genre, country of origin). We also compare these patterns to those observed when using the profiles in a downstream recommendation task. Our findings highlight both the potential and limitations of scrutable, LLM-based profiling in personalized systems.