ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpus

TL;DR

ARC-GEN is an open-source procedural generator that expands the ARC-AGI dataset, enabling better evaluation of skill acquisition efficiency in AI systems by faithfully mimicking the original tasks and supporting benchmark verification.

Contribution

It introduces a comprehensive, mimetic generator for the ARC-AGI dataset, covering all tasks and aiding in the development of a static benchmark suite.

Findings

Generator covers all 400 tasks in ARC-AGI.

It faithfully reproduces the distributional properties of the original dataset.

Supports verification of program correctness for AI challenge benchmarks.

Abstract

The Abstraction and Reasoning Corpus remains one of the most compelling and challenging benchmarks for tracking progress toward achieving Artificial General Intelligence. In contrast to other evaluation datasets designed to assess an agent's task-specific skills or accumulated knowledge, the ARC-AGI suite is specifically targeted at measuring skill acquisition efficiency, a trait that has (so far) been lacking in even the most sophisticated machine learning systems. For algorithms that require extensive intra-task exemplars, a significant constraint imposed by ARC-AGI is the modest cardinality of its demonstration set, comprising a small number of $\langle$ input, output $\rangle$ grids per task specifying the corresponding transformation. To embellish the space of viable sample pairs, this paper introduces ARC-GEN, an open-source procedural generator aimed at extending the original ARC-AGI training dataset as faithfully as possible. Unlike prior efforts, our generator is both exhaustive (covering all four-hundred tasks) and mimetic (more closely honoring the distributional properties and characteristics embodied in the initial ARC-AGI-1 release). We also discuss the use of this generator in establishing a static benchmark suite to verify the correctness of programs submitted to the 2025 Google Code Golf Championship.