AGI-Elo: How Far Are We From Mastering A Task?

TL;DR

This paper presents a unified, difficulty-aware rating system for evaluating AI models and humans across multiple domains, providing detailed insights into progress and remaining challenges on the path to AGI.

Contribution

It introduces a novel rating framework that jointly models task difficulty and model competency, enabling fine-grained evaluation beyond traditional metrics.

Findings

The system effectively captures the difficulty distribution of real-world challenges.

It provides interpretable insights into model progression and remaining gaps.

Validated across multiple datasets and domains.

Abstract

As the field progresses toward Artificial General Intelligence (AGI), there is a pressing need for more comprehensive and insightful evaluation frameworks that go beyond aggregate performance metrics. This paper introduces a unified rating system that jointly models the difficulty of individual test cases and the competency of AI models (or humans) across vision, language, and action domains. Unlike existing metrics that focus solely on models, our approach allows for fine-grained, difficulty-aware evaluations through competitive interactions between models and tasks, capturing both the long-tail distribution of real-world challenges and the competency gap between current models and full task mastery. We validate the generalizability and robustness of our system through extensive experiments on multiple established datasets and models across distinct AGI domains. The resulting rating distributions offer novel perspectives and interpretable insights into task difficulty, model progression, and the outstanding challenges that remain on the path to achieving full AGI task mastery.