Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling

TL;DR

This paper investigates the trade-off in RLHF-aligned language models between world modeling and agent acting capabilities, revealing that optimizing for one often diminishes the other, especially in long-form text generation.

Contribution

It empirically demonstrates the trade-off between world modeling and agent acting in RLHF models and proposes a potential explanation involving implicit blueprints limiting randomness.

Findings

RLHF models focus probability on anchor spans, reducing diversity.

A trade-off exists between prediction accuracy and coherent long-form generation.

Alignment techniques may inherently limit a model's predictive flexibility.

Abstract

RLHF-aligned LMs have shown unprecedented ability on both benchmarks and long-form text generation, yet they struggle with one foundational task: next-token prediction. As RLHF models become agent models aimed at interacting with humans, they seem to lose their world modeling -- the ability to predict what comes next in arbitrary documents, which is the foundational training objective of the Base LMs that RLHF adapts. Besides empirically demonstrating this trade-off, we propose a potential explanation: to perform coherent long-form generation, RLHF models restrict randomness via implicit blueprints. In particular, RLHF models concentrate probability on sets of anchor spans that co-occur across multiple generations for the same prompt, serving as textual scaffolding but also limiting a model's ability to generate documents that do not include these spans. We study this trade-off on the most effective current agent models, those aligned with RLHF, while exploring why this may remain a fundamental trade-off between models that act and those that predict, even as alignment techniques improve.