Re:Frame -- Retrieving Experience From Associative Memory

TL;DR

Re:Frame introduces an external associative memory module that retrieves expert experience to significantly enhance offline reinforcement learning performance from scarce and low-quality data without altering the core policy architecture.

Contribution

The paper presents Re:Frame, a novel plug-in module that leverages a small external memory to incorporate expert trajectories into offline RL, improving performance with minimal expert data.

Findings

Re:Frame improves performance on D4RL MuJoCo tasks with as few as 60 expert trajectories.

Using Re:Frame yields up to +10.7 normalized points over baseline.

The method requires no environment interaction or architecture modifications.

Abstract

Offline reinforcement learning (RL) often deals with suboptimal data when collecting large expert datasets is unavailable or impractical. This limitation makes it difficult for agents to generalize and achieve high performance, as they must learn primarily from imperfect or inconsistent trajectories. A central challenge is therefore how to best leverage scarce expert demonstrations alongside abundant but lower-quality data. We demonstrate that incorporating even a tiny amount of expert experience can substantially improve RL agent performance. We introduce Re:Frame (Retrieving Experience From Associative Memory), a plug-in module that augments a standard offline RL policy (e.g., Decision Transformer) with a small external Associative Memory Buffer (AMB) populated by expert trajectories drawn from a separate dataset. During training on low-quality data, the policy learns to retrieve expert data from the Associative Memory Buffer (AMB) via content-based associations and integrate them into decision-making; the same AMB is queried at evaluation. This requires no environment interaction and no modifications to the backbone architecture. On D4RL MuJoCo tasks, using as few as 60 expert trajectories (0.1% of a 6000-trajectory dataset), Re:Frame consistently improves over a strong Decision Transformer baseline in three of four settings, with gains up to +10.7 normalized points. These results show that Re:Frame offers a simple and data-efficient way to inject scarce expert knowledge and substantially improve offline RL from low-quality datasets.