Navigating Human Language Models with Synthetic Agents

TL;DR

This paper explores how large language models like GPT-2 can be used as sociological tools by training them on historical chess data and analyzing the generated text to reflect human beliefs and patterns in chess play.

Contribution

The study demonstrates that GPT-2 can accurately reflect human chess strategies and create latent representations of the game, offering a novel sociological approach using synthetic agents.

Findings

Model-generated move patterns align with human chess strategies

The model accurately reconstructs the chessboard state from text

Trajectory plots reveal legal move sequences across the board

Abstract

Modern natural language models such as the GPT-2/GPT-3 contain tremendous amounts of information about human belief in a consistently testable form. If these models could be shown to accurately reflect the underlying beliefs of the human beings that produced the data used to train these models, then such models become a powerful sociological tool in ways that are distinct from traditional methods, such as interviews and surveys. In this study, We train a version of the GPT-2 on a corpora of historical chess games, and then "launch" clusters of synthetic agents into the model, using text strings to create context and orientation. We compare the trajectories contained in the text generated by the agents/model and compare that to the known ground truth of the chess board, move legality, and historical patterns of play. We find that the percentages of moves by piece using the model are substantially similar from human patterns. We further find that the model creates an accurate latent representation of the chessboard, and that it is possible to plot trajectories of legal moves across the board using this knowledge.