Capturing Sparks of Abstraction for the ARC Challenge

TL;DR

This paper explores extracting high-level abstractions from LLM-generated code solutions to improve problem understanding in the ARC Challenge, providing open-source tools and data for future research.

Contribution

It introduces a method to extract 'sparks of abstraction' from LLM code solutions, enhancing understanding beyond surface-level outputs.

Findings

Extracted reusable problem-solving tactics from LLM code

Demonstrated potential for downstream tasks using abstractions

Provided open-source framework and data for further research

Abstract

Excellent progress has been made recently in solving ARC Challenge problems. However, it seems that new techniques may be required to push beyond 60% accuracy. Even commercial Large Language Models (LLMs) struggle to 'understand' many of the problems (when given the input and output grids), which makes discovering solutions by LLM-lead program search somewhat futile. In this work, LLM 'understanding' is attempted from a stronger starting position : An LLM is given complete solutions to tasks in code, and then asked to explain how the task is being solved at various levels of abstraction. Specifically, the LLM was given code solutions implemented in arc-dsl-llm (an LLM-legible version of Hodel's arc-dsl to obtain: (a) commented code; (b) code refactored into reusable functional chunks; (c) problem solution steps; and (d) high-level problem-solving tactics. We demonstrate that 'Sparks of Abstraction' can be extracted from the LLM output - in a form that could be used in downstream tasks with Local LLMs eligible to enter the ARC Prize. Both the arc-dsl-llm DSL framework (with the re-engineered solutions) and the Gemini LLM-generated data (along with the generation code) are made Open Source.