Task-Aware Automated User Profile Generation for Recommendation Simulation Using Large Language Models

TL;DR

This paper introduces APG4RecSim, a framework that automatically generates realistic user profiles for recommendation system simulations using large language models, improving performance and robustness.

Contribution

It presents a novel automated profile generation method that enhances simulation realism and generalizability, addressing limitations of manual profile crafting.

Findings

Achieves up to 7% improvement in ranking quality (nDCG@10)

Reduces rating distribution divergence by 8% (JSD)

Profiles are resilient to biases and stable across datasets and models

Abstract

Large Language Model (LLM)-based agent simulation has emerged as a promising approach to meet the increasing demand for real-time and rigorous evaluation in modern recommender systems. A typical LLM-driven simulation framework comprises three essential components: the profile module, memory module, and action module. However, existing studies have primarily concentrated on enhancing the memory and action modules, with limited attention to profile generation, which plays a pivotal role in ensuring realistic agent behaviours and aligning simulated interactions with real user dynamics. Moreover, the scarcity of datasets specifically designed for recommendation simulations has led to heavy reliance on manually crafted profiles, significantly limiting the scalability and generalisability of simulation frameworks across different datasets. To address these challenges, this work proposes an Automated Profile Generation Framework for Recommendation Simulation, APG4RecSim, that constructs realistic, coherent, and robust user profiles with minimal supervision. Extensive experiments on three benchmark datasets demonstrate that APG4RecSim achieves the best overall performance on discrimination, ranking, and rating tasks, improving ranking quality by up to 7% in nDCG@10 and reducing rating distribution divergence by 8% in JSD compared to existing profile-generation baselines. Beyond overall performance gains, our results show that profiles generated by APG4RecSim are resilient to popularity- and position-induced biases and maintain stable performance across datasets and different LLMs.