Emergent World Beliefs: Exploring Transformers in Stochastic Games
Adam Kamel, Tanish Rastogi, Michael Ma, Kailash Ranganathan, Kevin Zhu

TL;DR
This paper investigates how transformer-based large language models can develop internal representations of stochastic environments, like poker, capturing both deterministic and probabilistic aspects without explicit training for these features.
Contribution
The study extends understanding of emergent world models in LLMs to incomplete information domains, demonstrating their ability to learn and represent stochastic environment features.
Findings
Models learn hand ranks and equity without explicit instruction
Internal representations correlate with theoretical belief states
Probes show representations are decodeable and meaningful
Abstract
Transformer-based large language models (LLMs) have demonstrated strong reasoning abilities across diverse fields, from solving programming challenges to competing in strategy-intensive games such as chess. Prior work has shown that LLMs can develop emergent world models in games of perfect information, where internal representations correspond to latent states of the environment. In this paper, we extend this line of investigation to domains of incomplete information, focusing on poker as a canonical partially observable Markov decision process (POMDP). We pretrain a GPT-style model on Poker Hand History (PHH) data and probe its internal activations. Our results demonstrate that the model learns both deterministic structure, such as hand ranks, and stochastic features, such as equity, without explicit instruction. Furthermore, by using primarily nonlinear probes, we demonstrated that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsArtificial Intelligence in Games · Sports Analytics and Performance · Reinforcement Learning in Robotics
