Quantifying artificial intelligence through algorithmic generalization

TL;DR

This paper proposes a new theoretical framework based on algebraic circuit complexity to quantify and analyze the algorithmic reasoning capabilities and generalization of AI systems, addressing a critical gap in AI evaluation.

Contribution

It introduces algebraic circuit complexity as a formal tool to measure AI's algorithmic generalization, bridging computational complexity theory with AI benchmarking.

Findings

Provides a formal framework for quantifying algorithmic reasoning in AI

Demonstrates how algebraic circuits can serve as experimental benchmarks

Addresses key challenges in applying complexity theory to AI science

Abstract

The rapid development of artificial intelligence (AI) systems has created an urgent need for their scientific quantification. While their fluency across a variety of domains is impressive, AI systems fall short on tests requiring algorithmic reasoning -- a glaring limitation given the necessity for interpretable and reliable technology. Despite a surge of reasoning benchmarks emerging from the academic community, no theoretical framework exists to quantify algorithmic reasoning in AI systems. Here, we adopt a framework from computational complexity theory to quantify algorithmic generalization using algebraic expressions: algebraic circuit complexity. Algebraic circuit complexity theory -- the study of algebraic expressions as circuit models -- is a natural framework to study the complexity of algorithmic computation. Algebraic circuit complexity enables the study of generalization by defining benchmarks in terms of the computational requirements to solve a problem. Moreover, algebraic circuits are generic mathematical objects; an arbitrarily large number of samples can be generated for a specified circuit, making it an ideal experimental sandbox for the data-hungry models that are used today. In this Perspective, we adopt tools from algebraic circuit complexity, apply them to formalize a science of algorithmic generalization, and address key challenges for its successful application to AI science.