Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective

TL;DR

This paper introduces a novel product of experts approach using large language models to improve abstract reasoning on the ARC-AGI benchmark, combining data augmentation, diverse candidate generation, and LLM-based scoring.

Contribution

It presents a transparent, reproducible method that leverages task-specific data augmentation and LLM scoring to achieve state-of-the-art performance on ARC-AGI.

Findings

Achieved 71.6% solved tasks on ARC-AGI

Low inference cost of around 2 cents per task

Outperforms many existing approaches in transparency and efficiency

Abstract

The Abstraction and Reasoning Corpus (ARC-AGI) poses a significant challenge for large language models (LLMs), exposing limitations in their abstract reasoning abilities. In this work, we leverage task-specific data augmentations throughout the training, generation, and scoring phases, and employ a depth-first search algorithm to generate diverse, high-probability candidate solutions. Furthermore, we utilize the LLM not only as a generator but also as a scorer, using its output probabilities to select the most promising solutions. Our method achieves a score of 71.6% (286.5/400 solved tasks) on the public ARC-AGI evaluation set, demonstrating state-of-the-art performance among publicly available approaches. While concurrent closed-source work has reported higher scores, our method distinguishes itself through its transparency, reproducibility, and remarkably low inference cost, averaging only around 2ct per task on readily available hardware (we assume a price of 36ct/hour for a Nvidia 4090 GPU).