Eliciting the Priors of Large Language Models using Iterated In-Context Learning

TL;DR

This paper introduces a prompt-based method using iterated in-context learning to extract Bayesian priors from large language models, aligning them with human priors and applying to speculative future events.

Contribution

The paper presents a novel workflow for eliciting LLM priors via iterated in-context learning, enabling analysis of their implicit knowledge and comparison with human priors.

Findings

GPT-4 priors qualitatively match human priors in causal and proportion estimation.

The method successfully elicits priors for speculative future events.

The approach supports sampling from LLMs' implicit prior distributions.

Abstract

As Large Language Models (LLMs) are increasingly deployed in real-world settings, understanding the knowledge they implicitly use when making decisions is critical. One way to capture this knowledge is in the form of Bayesian prior distributions. We develop a prompt-based workflow for eliciting prior distributions from LLMs. Our approach is based on iterated learning, a Markov chain Monte Carlo method in which successive inferences are chained in a way that supports sampling from the prior distribution. We validated our method in settings where iterated learning has previously been used to estimate the priors of human participants -- causal learning, proportion estimation, and predicting everyday quantities. We found that priors elicited from GPT-4 qualitatively align with human priors in these settings. We then used the same method to elicit priors from GPT-4 for a variety of speculative events, such as the timing of the development of superhuman AI.