Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents

TL;DR

TheoryCoder-2 advances hierarchical planning in program-synthesis agents by enabling LLMs to learn abstractions from experience, significantly improving sample efficiency and problem-solving capabilities across diverse environments.

Contribution

It introduces a novel TBRL agent that actively learns abstractions using LLMs, reducing reliance on human-provided abstractions and enhancing generalization in hierarchical planning.

Findings

TheoryCoder-2 outperforms baseline agents in sample efficiency.

It solves complex tasks that prior methods cannot.

Requires minimal human prompts for abstraction learning.

Abstract

Humans learn abstractions and use them to plan efficiently to quickly generalize across tasks -- an ability that remains challenging for state-of-the-art large language model (LLM) agents and deep reinforcement learning (RL) systems. Inspired by the cognitive science of how people form abstractions and intuitive theories of their world knowledge, Theory-Based RL (TBRL) systems, such as TheoryCoder, exhibit strong generalization through effective use of abstractions. However, they heavily rely on human-provided abstractions and sidestep the abstraction-learning problem. We introduce TheoryCoder-2, a new TBRL agent that leverages LLMs' in-context learning ability to actively learn reusable abstractions rather than relying on hand-specified ones, by synthesizing abstractions from experience and integrating them into a hierarchical planning process. We conduct experiments on diverse environments, including BabyAI, Minihack and VGDL games like Sokoban. We find that TheoryCoder-2 is significantly more sample-efficient than baseline LLM agents augmented with classical planning domain construction, reasoning-based planning, and prior program-synthesis agents such as WorldCoder. TheoryCoder-2 is able to solve complex tasks that the baselines fail, while only requiring minimal human prompts, unlike prior TBRL systems.