Reinforcing Human Behavior Simulation via Verbal Feedback

TL;DR

This paper introduces DITTO, a reinforcement learning approach that uses verbal feedback to enhance LLMs in simulating human-like social behaviors across diverse tasks.

Contribution

The paper presents DITTO, a novel RL method that incorporates verbal feedback as a primary signal and introduces SOUL, a comprehensive benchmark for human-like behavior simulation.

Findings

DITTO achieves 36% improvement over the base model.

Exceeds GPT-5.4 on 6 out of 10 SOUL benchmarks.

Demonstrates RL with verbal feedback as a promising training direction.

Abstract

Humans learn social norms and behaviors from verbal feedback (e.g., a parent saying "that was rude" or a friend explaining "here's why that hurt"). Yet, learning from feedback for LLMs has largely focused on domains like code and math, where RL rewards are directly verifiable and condensed into scalar values. As LLMs are increasingly used to simulate human behavior, e.g., standing in for users, patients, students, and other personas, there is a pressing need to make them more human-like, which requires embracing a fundamentally different kind of signal: feedback that is verbal, subjective, and multi-faceted. We present DITTO, a model trained by treating verbal feedback as a first-class signal in reinforcement learning. After each rollout, DITTO receives verbal feedback and generates a feedback-conditioned improved rollout; both outputs are jointly optimized with GRPO, distilling verbal guidance into the base policy without requiring feedback at test time. We also introduce SOUL (Simulation gym Of hUman-Like behavior), a unified benchmark and training data suite spanning 10 tasks across six categories: Theory of Mind, character role play, social skill, learner simulation, user simulation, and persona simulation. DITTO achieves an average 36% improvement over the base model and exceeds GPT-5.4 on 6 of 10 SOUL benchmarks, demonstrating that RL with verbal feedback is a promising direction for training LLMs to simulate human behavior.