From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark

TL;DR

This paper evaluates reasoning-oriented large language models on the ARC benchmark, introduces a knowledge augmentation method called KAAR that improves reasoning and generalization, and demonstrates significant performance gains over existing methods.

Contribution

The paper proposes KAAR, a novel knowledge augmentation framework that enhances LLM reasoning by incorporating hierarchical priors, leading to improved performance on the ARC benchmark.

Findings

KAAR outperforms non-augmented methods with around 5% absolute gains.

Repeated-sampling planning-aided code generation (RSPC) achieves high accuracy and generalization.

ARC remains a challenging benchmark for reasoning-oriented LLMs.

Abstract

Recent reasoning-oriented LLMs have demonstrated strong performance on challenging tasks such as mathematics and science examinations. However, core cognitive faculties of human intelligence, such as abstract reasoning and generalization, remain underexplored. To address this, we evaluate recent reasoning-oriented LLMs on the Abstraction and Reasoning Corpus (ARC) benchmark, which explicitly demands both faculties. We formulate ARC as a program synthesis task and propose nine candidate solvers. Experimental results show that repeated-sampling planning-aided code generation (RSPC) achieves the highest test accuracy and demonstrates consistent generalization across most LLMs. To further improve performance, we introduce an ARC solver, Knowledge Augmentation for Abstract Reasoning (KAAR), which encodes core knowledge priors within an ontology that classifies priors into three hierarchical levels based on their dependencies. KAAR progressively expands LLM reasoning capacity by gradually augmenting priors at each level, and invokes RSPC to generate candidate solutions after each augmentation stage. This stage-wise reasoning reduces interference from irrelevant priors and improves LLM performance. Empirical results show that KAAR maintains strong generalization and consistently outperforms non-augmented RSPC across all evaluated LLMs, achieving around 5% absolute gains and up to 64.52% relative improvement. Despite these achievements, ARC remains a challenging benchmark for reasoning-oriented LLMs, highlighting future avenues of progress in LLMs.