CreAgent: Towards Long-Term Evaluation of Recommender System under Platform-Creator Information Asymmetry

TL;DR

CreAgent introduces a novel LLM-based creator simulation framework that models long-term recommender system impacts considering information asymmetry between platforms and creators, enhancing evaluation reliability.

Contribution

The paper presents CreAgent, a new LLM-powered creator simulation agent that incorporates game theory and cognitive frameworks to better model long-term effects in recommender systems under information asymmetry.

Findings

CreAgent aligns well with real-world creator behaviors.

Simulation results show improved long-term evaluation accuracy.

Fairness and diversity algorithms benefit from the CreAgent simulation platform.

Abstract

Ensuring the long-term sustainability of recommender systems (RS) emerges as a crucial issue. Traditional offline evaluation methods for RS typically focus on immediate user feedback, such as clicks, but they often neglect the long-term impact of content creators. On real-world content platforms, creators can strategically produce and upload new items based on user feedback and preference trends. While previous studies have attempted to model creator behavior, they often overlook the role of information asymmetry. This asymmetry arises because creators primarily have access to feedback on the items they produce, while platforms possess data on the entire spectrum of user feedback. Current RS simulators, however, fail to account for this asymmetry, leading to inaccurate long-term evaluations. To address this gap, we propose CreAgent, a Large Language Model (LLM)-empowered creator simulation agent. By incorporating game theory's belief mechanism and the fast-and-slow thinking framework, CreAgent effectively simulates creator behavior under conditions of information asymmetry. Additionally, we enhance CreAgent's simulation ability by fine-tuning it using Proximal Policy Optimization (PPO). Our credibility validation experiments show that CreAgent aligns well with the behaviors between real-world platform and creator, thus improving the reliability of long-term RS evaluations. Moreover, through the simulation of RS involving CreAgents, we can explore how fairness- and diversity-aware RS algorithms contribute to better long-term performance for various stakeholders. CreAgent and the simulation platform are publicly available at https://github.com/shawnye2000/CreAgent.