Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation

TL;DR

This paper introduces ContextSim, a framework using large language models to simulate realistic user behavior in recommender system evaluation by incorporating contextual daily life activities.

Contribution

It presents a novel LLM-based agent framework that models user context and internal thoughts, improving the realism of user simulations for recommender system testing.

Findings

ContextSim generates user interactions more aligned with human behavior.

Optimizing RS parameters with ContextSim improves real-world engagement.

The approach enhances offline evaluation correlation with online performance.

Abstract

Recommender systems are central to online services, enabling users to navigate through massive amounts of content across various domains. However, their evaluation remains challenging due to the disconnect between offline metrics and online performance. The emergence of Large Language Model-powered agents offers a promising solution, yet existing studies model users in isolation, neglecting the contextual factors such as time, location, and needs, which fundamentally shape human decision-making. In this paper, we introduce ContextSim, an LLM agent framework that simulates believable user proxies by anchoring interactions in daily life activities. Namely, a life simulation module generates scenarios specifying when, where, and why users engage with recommendations. To align preferences with genuine humans, we model agents' internal thoughts and enforce consistency at both the action and trajectory levels. Experiments across domains show our method generates interactions more closely aligned with human behavior than prior work. We further validate our approach through offline A/B testing correlation and show that RS parameters optimized using ContextSim yield improved real-world engagement.