From Model Choice to Model Belief: Establishing a New Measure for LLM-Based Research

TL;DR

This paper introduces 'model belief,' a new measure derived from LLM token probabilities, which is more statistically efficient and effective for research applications than traditional model choice data.

Contribution

It formalizes and proves the properties of model belief, demonstrating its advantages over model choice in efficiency and accuracy for LLM-based research.

Findings

Model belief is asymptotically equivalent to mean model choice.

Model belief has lower variance and faster convergence.

Using model belief reduces computational costs significantly.

Abstract

Large language models (LLMs) are increasingly used to simulate human behavior, but common practices to use LLM-generated data are inefficient. Treating an LLM's output ("model choice") as a single data point underutilizes the information inherent to the probabilistic nature of LLMs. This paper introduces and formalizes "model belief," a measure derived from an LLM's token-level probabilities that captures the model's belief distribution over choice alternatives in a single generation run. The authors prove that model belief is asymptotically equivalent to the mean of model choices (a non-trivial property) but forms a more statistically efficient estimator, with lower variance and a faster convergence rate. Analogous properties are shown to hold for smooth functions of model belief and model choice often used in downstream applications. The authors demonstrate the performance of model belief through a demand estimation study, where an LLM simulates consumer responses to different prices. In practical settings with limited numbers of runs, model belief explains and predicts ground-truth model choice better than model choice itself, and reduces the computation needed to reach sufficiently accurate estimates by roughly a factor of 20. The findings support using model belief as the default measure to extract more information from LLM-generated data.