Learning Hierarchical Procedural Memory for LLM Agents through Bayesian Selection and Contrastive Refinement

TL;DR

MACLA introduces a hierarchical procedural memory system that enables large language models to adapt and improve through external memory, Bayesian selection, and contrastive refinement, achieving high performance and efficiency across multiple benchmarks.

Contribution

The paper presents MACLA, a novel framework that decouples reasoning from learning in LLMs, using external hierarchical memory and Bayesian methods for efficient adaptation without updating model parameters.

Findings

Achieves 78.1% average performance across four benchmarks.

Reaches 90.3% on unseen ALFWorld tasks with 3.1% positive generalization.

Constructs memory 2800 times faster than parameter-training baselines.

Abstract

We present MACLA, a framework that decouples reasoning from learning by maintaining a frozen large language model while performing all adaptation in an external hierarchical procedural memory. MACLA extracts reusable procedures from trajectories, tracks reliability via Bayesian posteriors, selects actions through expected-utility scoring, and refines procedures by contrasting successes and failures. Across four benchmarks (ALFWorld, WebShop, TravelPlanner, InterCodeSQL), MACLA achieves 78.1 percent average performance, outperforming all baselines. On ALFWorld unseen tasks, MACLA reaches 90.3 percent with 3.1 percent positive generalization. The system constructs memory in 56 seconds, 2800 times faster than the state-of-the-art LLM parameter-training baseline, compressing 2851 trajectories into 187 procedures. Experimental results demonstrate that structured external memory with Bayesian selection and contrastive refinement enables sample-efficient, interpretable, and continually improving agents without LLM parameter updates.