Structured Program Synthesis using LLMs: Results and Insights from the IPARC Challenge

TL;DR

This paper evaluates LLMs for structured program synthesis using the IPARC challenge, revealing key insights into effective collaboration and strategies for improving automated code generation in complex tasks.

Contribution

It introduces a structured inductive programming approach with LLMs that successfully addresses all IPARC challenge categories, providing new insights into human-LLM collaboration.

Findings

Prior structuring improves LLM performance

LLMs can assist in structuring but need human refinement

Code reuse and freezing correct code enhance efficiency

Abstract

The IPARC Challenge, inspired by ARC, provides controlled program synthesis tasks over synthetic images to evaluate automatic program construction, focusing on sequence, selection, and iteration. This set of 600 tasks has resisted automated solutions. This paper presents a structured inductive programming approach with LLMs that successfully solves tasks across all IPARC categories. The controlled nature of IPARC reveals insights into LLM-based code generation, including the importance of prior structuring, LLMs' ability to aid structuring (requiring human refinement), the need to freeze correct code, the efficiency of code reuse, and how LLM-generated code can spark human creativity. These findings suggest valuable mechanisms for human-LLM collaboration in tackling complex program synthesis.