Playing Large Games with Oracles and AI Debate

TL;DR

This paper introduces an efficient oracle-based algorithm for regret minimization in large, language-based repeated games, with applications to AI Safety via Debate, achieving low regret dependence on the number of actions.

Contribution

It presents a novel algorithm that minimizes internal and external regret simultaneously with logarithmic dependence on actions, advancing large-scale game playing methods.

Findings

Algorithm achieves logarithmic regret dependence on actions.

Experiments demonstrate benefits in AI Safety via Debate.

Efficient regret minimization in large, language-based games.

Abstract

We consider regret minimization in repeated games with a very large number of actions. Such games are inherent in the setting of AI Safety via Debate \cite{irving2018ai}, and more generally games whose actions are language-based. Existing algorithms for online game playing require per-iteration computation polynomial in the number of actions, which can be prohibitive for large games. We thus consider oracle-based algorithms, as oracles naturally model access to AI agents. With oracle access, we characterize when internal and external regret can be minimized efficiently. We give a novel efficient algorithm for simultaneous external and internal regret minimization whose regret depends logarithmically on the number of actions. We conclude with experiments in the setting of AI Safety via Debate that shows the benefit of insights from our algorithmic analysis.