ReGAL: Refactoring Programs to Discover Generalizable Abstractions

TL;DR

ReGAL is a gradient-free method that refactors code to learn reusable abstractions, improving program prediction accuracy across diverse domains by enabling large language models to utilize shared function libraries.

Contribution

ReGAL introduces a novel refactoring-based approach for learning reusable code abstractions that enhance LLM program synthesis without gradient updates.

Findings

ReGAL improves accuracy of LLMs on multiple datasets.

Shared abstractions facilitate better program prediction.

ReGAL outperforms some existing methods on key benchmarks.

Abstract

While large language models (LLMs) are increasingly being used for program synthesis, they lack the global view needed to develop useful abstractions; they generally predict programs one at a time, often repeating the same functionality. Generating redundant code from scratch is both inefficient and error-prone. To address this, we propose Refactoring for Generalizable Abstraction Learning (ReGAL), a gradient-free method for learning a library of reusable functions via code refactorization, i.e., restructuring code without changing its execution output. ReGAL learns from a small set of existing programs, iteratively verifying and refining its abstractions via execution. We find that the shared function libraries discovered by ReGAL make programs easier to predict across diverse domains. On five datasets -- LOGO graphics generation, Date reasoning, TextCraft (a Minecraft-based text-game) MATH, and TabMWP -- both open-source and proprietary LLMs improve in accuracy when predicting programs with ReGAL functions. For CodeLlama-13B, ReGAL results in absolute accuracy increases of 11.5% on LOGO, 26.1% on date understanding, and 8.1% on TextCraft, outperforming GPT-3.5 in two of three domains. Our analysis reveals ReGAL's abstractions encapsulate frequently-used subroutines as well as environment dynamics.