Understanding and Benchmarking Artificial Intelligence: OpenAI's o3 Is Not AGI

TL;DR

OpenAI's o3 achieves high scores on a specific benchmark by extensive trialling, but this does not demonstrate true artificial general intelligence, highlighting the need for new benchmarks and approaches for assessing genuine intelligence.

Contribution

The paper introduces a new understanding of intelligence based on goal diversity and efficiency, critiques the limitations of current benchmarks like ARC-AGI, and proposes a more comprehensive benchmark for true AGI assessment.

Findings

o3's high score results from extensive trialling of predefined operations

Current benchmarks do not adequately measure general intelligence

New benchmarks are needed to evaluate diverse, unknown problem-solving abilities

Abstract

OpenAI's o3 achieves a high score of 87.5 % on ARC-AGI, a benchmark proposed to measure intelligence. This raises the question whether systems based on Large Language Models (LLMs), particularly o3, demonstrate intelligence and progress towards artificial general intelligence (AGI). Building on the distinction between skills and intelligence made by Fran\c{c}ois Chollet, the creator of ARC-AGI, a new understanding of intelligence is introduced: an agent is the more intelligent, the more efficiently it can achieve the more diverse goals in the more diverse worlds with the less knowledge. An analysis of the ARC-AGI benchmark shows that its tasks represent a very specific type of problem that can be solved by massive trialling of combinations of predefined operations. This method is also applied by o3, achieving its high score through the extensive use of computing power. However, for most problems in the physical world and in the human domain, solutions cannot be tested in advance and predefined operations are not available. Consequently, massive trialling of predefined operations, as o3 does, cannot be a basis for AGI - instead, new approaches are required that can reliably solve a wide variety of problems without existing skills. To support this development, a new benchmark for intelligence is outlined that covers a much higher diversity of unknown tasks to be solved, thus enabling a comprehensive assessment of intelligence and of progress towards AGI.