Capturing Sparks of Abstraction for the ARC Challenge
Martin Andrews

TL;DR
This paper explores extracting high-level abstractions from LLM-generated code solutions to improve problem understanding in the ARC Challenge, providing open-source tools and data for future research.
Contribution
It introduces a method to extract 'sparks of abstraction' from LLM code solutions, enhancing understanding beyond surface-level outputs.
Findings
Extracted reusable problem-solving tactics from LLM code
Demonstrated potential for downstream tasks using abstractions
Provided open-source framework and data for further research
Abstract
Excellent progress has been made recently in solving ARC Challenge problems. However, it seems that new techniques may be required to push beyond 60% accuracy. Even commercial Large Language Models (LLMs) struggle to 'understand' many of the problems (when given the input and output grids), which makes discovering solutions by LLM-lead program search somewhat futile. In this work, LLM 'understanding' is attempted from a stronger starting position : An LLM is given complete solutions to tasks in code, and then asked to explain how the task is being solved at various levels of abstraction. Specifically, the LLM was given code solutions implemented in arc-dsl-llm (an LLM-legible version of Hodel's arc-dsl to obtain: (a) commented code; (b) code refactored into reusable functional chunks; (c) problem solution steps; and (d) high-level problem-solving tactics. We demonstrate that 'Sparks…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Machine Learning in Materials Science · Model-Driven Software Engineering Techniques
