Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)

TL;DR

This paper introduces BiasRecBench, a benchmark revealing that current LLM-based recommender agents are highly vulnerable to biases, which compromises their reliability in critical real-world tasks like research, e-commerce, and recruitment.

Contribution

The work presents BiasRecBench, a novel benchmark and synthesis pipeline to evaluate and demonstrate the susceptibility of LLM recommenders to biases, highlighting a key reliability challenge.

Findings

LLMs often succumb to injected biases despite reasoning capabilities.

BiasRecBench effectively exposes vulnerabilities in state-of-the-art LLMs.

Current LLM recommenders need better alignment strategies to mitigate bias risks.

Abstract

Current Large Language Models (LLMs) are gradually exploited in practically valuable agentic workflows such as Deep Research, E-commerce recommendation, and job recruitment. In these applications, LLMs need to select some optimal solutions from massive candidates, which we term as \textit{LLM-as-a-Recommender} paradigm. However, the reliability of using LLM agents for recommendations is underexplored. In this work, we introduce a \textbf{Bias} \textbf{Rec}ommendation \textbf{Bench}mark (\textbf{BiasRecBench}) to highlight the critical vulnerability of such agents to biases in high-value real-world tasks. The benchmark includes three practical domains: paper review, e-commerce, and job recruitment. We construct a \textsc{Bias Synthesis Pipeline with Calibrated Quality Margins} that 1) synthesizes evaluation data by controlling the quality gap between optimal and sub-optimal options to provide a calibrated testbed to elicit the vulnerability to biases; 2) injects contextual biases that are logical and suitable for option contexts. Extensive experiments on both SOTA (Gemini-{2.5,3}-pro, GPT-4o, DeepSeek-R1) and small-scale LLMs reveal that agents frequently succumb to injected biases despite having sufficient reasoning capabilities to identify the ground truth. These findings expose a significant reliability bottleneck in current agentic workflows, calling for specialized alignment strategies for LLM-as-a-Recommender. The complete code and evaluation datasets will be made publicly available shortly.