Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures

TL;DR

This paper introduces Cross-Entropy (Xent) Games as a framework to evaluate and measure the capabilities of Large Language Models through a variety of tasks beyond simple text generation, using game-theoretic principles.

Contribution

It formulates a novel class of tasks called Xent Games based on LLM probability measures, enabling new capability benchmarks and evaluation methods.

Findings

Xent Games encompass diverse tasks like summarization and debating.

They can be represented as simple computational graphs and programs.

The framework allows constructing comprehensive capability benchmarks.

Abstract

Large Language Models (LLMs) define probability measures on text. By considering the implicit knowledge question of what it means for an LLM to know such a measure and what it entails algorithmically, we are naturally led to formulate a series of tasks that go beyond generative sampling, involving forms of summarization, counterfactual thinking, anomaly detection, originality search, reverse prompting, debating, creative solving, etc. These tasks can be formulated as games based on LLM measures, which we call Cross-Entropy (Xent) Games. Xent Games can be single-player or multi-player. They involve cross-entropy scores and cross-entropy constraints, and can be expressed as simple computational graphs and programs. We show the Xent Game space is large enough to contain a wealth of interesting examples, while being constructible from basic game-theoretic consistency axioms. We then discuss how the Xent Game space can be used to measure the abilities of LLMs. This leads to the construction of Xent Game measures: finite families of Xent Games that can be used as capability benchmarks, built from a given scope, by extracting a covering measure. To address the unbounded scope problem associated with the challenge of measuring general abilities, we propose to explore the space of Xent Games in a coherent fashion, using ideas inspired by evolutionary dynamics.