ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)

TL;DR

ConceptSearch is a novel algorithm that uses LLMs and concept-based scoring to improve program search efficiency and accuracy on the challenging ARC benchmark, demonstrating significant performance gains.

Contribution

We introduce ConceptSearch, a new function-search algorithm combining LLMs with concept-based scoring to enhance program synthesis for complex generalization tasks.

Findings

Achieved significant performance improvement over GPT-4 prompting.

Concept-based scoring is up to 30% more efficient than Hamming distance.

Demonstrated effectiveness on the ARC challenge.

Abstract

The Abstraction and Reasoning Corpus (ARC) poses a significant challenge to artificial intelligence, demanding broad generalization and few-shot learning capabilities that remain elusive for current deep learning methods, including large language models (LLMs). While LLMs excel in program synthesis, their direct application to ARC yields limited success. To address this, we introduce ConceptSearch, a novel function-search algorithm that leverages LLMs for program generation and employs a concept-based scoring method to guide the search efficiently. Unlike simplistic pixel-based metrics like Hamming distance, ConceptSearch evaluates programs on their ability to capture the underlying transformation concept reflected in the input-output examples. We explore three scoring functions: Hamming distance, a CNN-based scoring function, and an LLM-based natural language scoring function. Experimental results demonstrate the effectiveness of ConceptSearch, achieving a significant performance improvement over direct prompting with GPT-4. Moreover, our novel concept-based scoring exhibits up to 30% greater efficiency compared to Hamming distance, measured in terms of the number of iterations required to reach the correct solution. These findings highlight the potential of LLM-driven program search when integrated with concept-based guidance for tackling challenging generalization problems like ARC.