The Generalized Turing Test: A Foundation for Comparing Intelligence

TL;DR

The paper proposes the Generalized Turing Test (GTT), a formal, task-agnostic framework for comparing agent intelligence through indistinguishability, supported by empirical evaluations on modern models.

Contribution

It introduces the GTT framework, formalizes conditions for agent comparison, and empirically demonstrates its effectiveness across various models and trials.

Findings

The GTT framework produces meaningful empirical orderings of agent capabilities.

Indistinguishability-based comparisons align with existing rankings.

The framework is adaptable with variants like querying and bounded interactions.

Abstract

We introduce the Generalized Turing Test (GTT), a formal framework for comparing the capabilities of arbitrary agents via indistinguishability. For agents A and B, we define the Turing comparator A $\geq$ B to hold if B, acting as a distinguisher, cannot reliably distinguish between interactions with A (instructed to imitate B) and another instance of B. This yields a dataset- and task-agnostic notion of relative intelligence. We study the comparator's structure, including conditions under which it is transitive and therefore induces an ordering over equivalence classes, and we define and analyze variants with querying, bounded interaction, and fixed distinguishers. To complement the theory, we instantiate the framework on a collection of modern models, empirically evaluating pairwise indistinguishability across thousands of trials. The resulting comparisons exhibit a stratified structure consistent with existing rankings, hinting that the proposed framework yields meaningful empirical orderings. Our results position indistinguishability as a unifying lens for reasoning about intelligence, suggesting a foundation for evaluation and, potentially, training objectives that are inherently independent of fixed datasets or benchmarks.