HumanLM: Simulating Users with State Alignment Beats Response Imitation

TL;DR

HumanLM introduces a novel user simulation framework that models underlying user states with reinforcement learning, significantly improving the realism and alignment of simulated responses compared to traditional imitation methods.

Contribution

The paper presents HumanLM, a new training framework that generates latent user states aligned with responses, enhancing user simulation accuracy beyond surface-level imitation.

Findings

HumanLM outperforms alternatives with 16.3% average improvement in alignment scores.

HumanLM achieves higher similarity to real user responses in real-time studies.

The Humanual benchmark provides extensive data for evaluating user simulators.

Abstract

Large Language Models (LLMs) are increasingly used to simulate how specific users respond to a given context, enabling more user-centric applications that rely on user feedback. However, existing user simulators mostly imitate surface-level patterns and language styles, which fail to reflect the underlying states of real users (e.g., beliefs and emotions). To address these limitations, we propose a novel training framework, HumanLM, which builds user simulators that accurately reflect real users. Our key insight is that, in addition to generating responses, the model should generate natural-language latent states that align with ground-truth responses through reinforcement learning. These latent states correspond to a set of psychologically grounded state dimensions that drive how real users respond. HumanLM further synthesizes these aligned latent states into responses that accurately represent real users. For extensive evaluation, we develop Humanual, a comprehensive benchmark for simulating real users based on public data. Humanual consists of six large-scale datasets with 26k users and 216k responses in total, spanning diverse tasks such as generating user responses to daily life issues, political blogs, and chat sessions with LLM assistants. Across datasets, HumanLM significantly outperforms alternative approaches, achieving an average relative improvement of 16.3% in alignment scores from an LLM judge. In a real-time simulation study with 111 participants, HumanLM achieves the highest similarity to real user responses and competitive human-likeness scores.