The Mouth is Not the Brain: Bridging Energy-Based World Models and Language Generation

TL;DR

This paper introduces a modular architecture that separates world modeling from language generation, demonstrating improved controllability and coherence in text output by connecting a domain-specific energy-based world model with a frozen language model.

Contribution

It proposes a novel framework that explicitly decouples world understanding from language modeling, enabling better control and coherence in generated text.

Findings

World model conditioning reduces cross-entropy and increases semantic similarity.

Energy function effectively distinguishes plausible from implausible configurations.

Causal interventions on attributes influence generated text in a statistically consistent manner.

Abstract

Large Language Models (LLMs) generate fluent text, yet whether they truly understand the world or merely produce plausible texts about it remains contested. We propose an architectural principle, the mouth is not the brain, that explicitly separates world models from language models. Our architecture comprises three components: a DBM that captures domain structure as an energy-based world model, an adapter that projects latent belief states into embedding space, and a frozen GPT-2 that provides linguistic competence without domain knowledge. We instantiate this framework in the consumer review domain using Amazon smartphone reviews. Experiments demonstrate that (1) world model conditioning achieves lower cross-entropy loss and higher semantic similarity than architectural baselines including direct projection and full fine-tuning, while qualitative analysis reveals that soft prompt conditioning resolves a trade-off that prompt-based approaches cannot: simple prompts lack expressiveness while detailed prompts cause output collapse in small LLMs; (2) the DBM's energy function distinguishes coherent from incoherent market configurations, assigning higher energy to implausible brand-price combinations; and (3) interventions on specific attributes propagate causally to generated text with intervened outputs exhibiting distributions statistically consistent with naturally occurring samples sharing the target configuration. These findings suggest that even small-scale language models can achieve consistent, controllable generation when connected to an appropriate world model, providing empirical support for separating linguistic competence from world understanding.