RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation

TL;DR

This paper introduces RLHS, a novel approach that mitigates misalignment in RLHF by using hindsight simulation, leading to improved alignment and robustness across various tasks and evaluation methods.

Contribution

The paper proposes RLHS, a new method that uses simulated outcomes to decouple feedback from AI predictions, effectively reducing misalignment in reinforcement learning from human feedback.

Findings

RLHS significantly outperforms RLHF in alignment tasks.

RLHS demonstrates robustness across multiple evaluation benchmarks.

Hindsight simulation reduces systematic misalignment in AI models.

Abstract

While Reinforcement Learning from Human Feedback (RLHF) has shown promise in aligning generative AI, we present empirical evidence that it can also cause severe, systematic misalignment. We hypothesize that this stems from evaluator feedback depending on downstream outcome predictions (foresight) that can be influenced by the AI's output, inducing Goodhart's law dynamics. We present a theoretical analysis showing that conditioning evaluator feedback on downstream observations (hindsight) inhibits this effect by decoupling the alignment signal from potentially compromised predictions--crucially, the result holds even if the observed outcomes are sampled from the AI's own world model. Building on this insight, we introduce Reinforcement Learning from Hindsight Simulation (RLHS), which presents plausible simulated outcomes to evaluators before eliciting feedback. We validate RLHS across three consultancy settings--marketplace interactions, restaurant recommendations, and online course advising--using both online (PPO) and offline (DPO) fine-tuning methods, and show that it substantially improves alignment over RLHF in experiments and human evaluations. We perform post-hoc benchmark evaluations on TruthfulQA, HaluEval, and TrustLLM, finding that even after single-task fine-tuning, RLHF misalignment persists, whereas RLHS consistently outperforms baselines and demonstrates robust alignment generalization. The project webpage and code are available at https://rl-hindsight.github.io.