Growing Through Experience: Scaling Episodic Grounding in Language Models

TL;DR

This paper introduces a scalable episodic learning framework that enhances language models' ability to leverage past experiences for improved physical planning, especially in medium-sized models, by combining structured experience collection and novel distillation techniques.

Contribution

A novel weak-to-strong episodic learning framework that transfers episodic behaviors from smaller to larger language models, improving scalability and task performance.

Findings

Outperforms state-of-the-art proprietary LMs by 3.45% on planning and QA tasks.

Significant improvements in deeper LM layers for task alignment.

Stable generalization to unseen scenarios with increased planning complexity.

Abstract

Language models (LMs) require robust episodic grounding-the capacity to learn from and apply past experiences-to excel at physical planning tasks. Current episodic grounding approaches struggle with scalability and integration, limiting their effectiveness, especially for medium-sized LMs (7B parameters). While larger LMs (70-405B parameters) possess superior hierarchical representations and extensive pre-trained knowledge, they encounter a fundamental scale paradox: despite their advanced abstraction capabilities, they lack efficient mechanisms to leverage experience streams. We propose a scalable weak-to-strong episodic learning framework that effectively transfers episodic behaviors from smaller to larger LMs. This framework integrates Monte Carlo tree search for structured experience collection with a novel distillation method, preserving the inherent LM capabilities while embedding episodic memory. Experiments demonstrate our method surpasses state-of-the-art proprietary LMs by 3.45% across diverse planning and question-answering tasks. Layer-wise probing further indicates significant improvements in task alignment, especially within deeper LM layers, highlighting stable generalization even for previously unseen scenarios with increased planning complexity-conditions where baseline methods degrade markedly.