Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation

TL;DR

This paper introduces a framework for long-horizon simulation of language model advice propagation to improve societal safety alignment, supported by a new dataset and significant performance gains on safety benchmarks.

Contribution

It presents a novel simulation-based approach for assessing long-term societal impacts of language model advice and introduces a new harm scenario dataset for safety evaluation.

Findings

Over 20% improvement on the new harm scenario dataset

Average win rate over 70% against strong baselines on existing benchmarks

Demonstrates potential for safer, more aligned language model agents

Abstract

Given the growing influence of language model-based agents on high-stakes societal decisions, from public policy to healthcare, ensuring their beneficial impact requires understanding the far-reaching implications of their suggestions. We propose a proof-of-concept framework that projects how model-generated advice could propagate through societal systems on a macroscopic scale over time, enabling more robust alignment. To assess the long-term safety awareness of language models, we also introduce a dataset of 100 indirect harm scenarios, testing models' ability to foresee adverse, non-obvious outcomes from seemingly harmless user prompts. Our approach achieves not only over 20% improvement on the new dataset but also an average win rate exceeding 70% against strong baselines on existing safety benchmarks (AdvBench, SafeRLHF, WildGuardMix), suggesting a promising direction for safer agents.