A Formal Solution to the Grain of Truth Problem

TL;DR

This paper provides a formal, general solution to the grain of truth problem in multi-agent Bayesian settings, demonstrating convergence of Thompson sampling to equilibrium in unknown environments.

Contribution

It constructs a comprehensive class of policies including all computable and Bayes-optimal policies, addressing longstanding limitations in multi-agent learning theory.

Findings

Constructs a policy class containing all computable and Bayes-optimal policies.

Proves convergence of Thompson sampling to ε-Nash equilibria in unknown environments.

Shows theoretical results can be approximated computationally.

Abstract

A Bayesian agent acting in a multi-agent environment learns to predict the other agents' policies if its prior assigns positive probability to them (in other words, its prior contains a \emph{grain of truth}). Finding a reasonably large class of policies that contains the Bayes-optimal policies with respect to this class is known as the \emph{grain of truth problem}. Only small classes are known to have a grain of truth and the literature contains several related impossibility results. In this paper we present a formal and general solution to the full grain of truth problem: we construct a class of policies that contains all computable policies as well as Bayes-optimal policies for every lower semicomputable prior over the class. When the environment is unknown, Bayes-optimal agents may fail to act optimally even asymptotically. However, agents based on Thompson sampling converge to play {\epsilon}-Nash equilibria in arbitrary unknown computable multi-agent environments. While these results are purely theoretical, we show that they can be computationally approximated arbitrarily closely.