Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMs

TL;DR

This paper introduces SEAM, a lightweight module that stores and generates structured experience entries to improve the reasoning accuracy of frozen LLMs with minimal overhead.

Contribution

SEAM is a novel, trainable plug-in that enhances frozen LLMs by generating utility-optimized structured experiences without retraining the entire model.

Findings

SEAM achieves consistent accuracy improvements on mathematical reasoning benchmarks.

SEAM introduces minimal computational overhead during inference.

Extensive analyses reveal the mechanisms behind SEAM's robustness and effectiveness.

Abstract

Large language models (LLMs) are largely static and often redo reasoning or repeat mistakes. Prior experience reuse typically relies on external retrieval, which is similarity-based, can introduce noise, and adds latency. We introduce SEAM (Structured Experience Adapter Module), a lightweight, executor-specific plug-in that stores experience in its parameters and generates a structured, instance-tailored experience entry in a single forward pass to guide a frozen LLM executor. SEAM is trained for utility via executor rollouts and GRPO while keeping the executor frozen, and it can be further improved after deployment with supervised fine-tuning on logged successful trajectories. Experiments on mathematical reasoning benchmarks show consistent accuracy gains across executors with low overhead. Extensive ablations and analyses further elucidate the mechanisms underlying SEAM's effectiveness and robustness.