Filtering Learning Histories Enhances In-Context Reinforcement Learning

TL;DR

This paper introduces a novel dataset preprocessing method called learning history filtering (LHF) that improves in-context reinforcement learning by filtering suboptimal learning histories, leading to more robust performance across various environments and algorithms.

Contribution

LHF is the first approach to prevent inheriting suboptimal behaviors in ICRL through dataset filtering, compatible with existing algorithms and effective in noisy data scenarios.

Findings

LHF improves performance on ICRL benchmarks.

LHF is robust across different hyperparameters and sampling strategies.

LHF performs especially well with noisy data.

Abstract

Transformer models (TMs) have exhibited remarkable in-context reinforcement learning (ICRL) capabilities, allowing them to generalize to and improve in previously unseen environments without re-training or fine-tuning. This is typically accomplished by imitating the complete learning histories of a source RL algorithm over a substantial amount of pretraining environments, which, however, may transfer suboptimal behaviors inherited from the source algorithm/dataset. Therefore, in this work, we address the issue of inheriting suboptimality from the perspective of dataset preprocessing. Motivated by the success of the weighted empirical risk minimization, we propose a simple yet effective approach, learning history filtering (LHF), to enhance ICRL by reweighting and filtering the learning histories based on their improvement and stability characteristics. To the best of our knowledge, LHF is the first approach to avoid source suboptimality by dataset preprocessing, and can be combined with the current state-of-the-art (SOTA) ICRL algorithms. We substantiate the effectiveness of LHF through a series of experiments conducted on the well-known ICRL benchmarks, encompassing both discrete environments and continuous robotic manipulation tasks, with three SOTA ICRL algorithms (AD, DPT, DICP) as the backbones. LHF exhibits robust performance across a variety of suboptimal scenarios, as well as under varying hyperparameters and sampling strategies. Notably, the superior performance of LHF becomes more pronounced in the presence of noisy data, indicating the significance of filtering learning histories.