Cross-Episodic Curriculum for Transformer Agents

TL;DR

The paper introduces Cross-Episodic Curriculum (CEC), a novel algorithm that enhances Transformer agents' learning efficiency and generalization by structuring cross-episodic experiences into a curriculum, demonstrated in multi-task reinforcement and imitation learning scenarios.

Contribution

This work presents CEC, a new curriculum learning method that leverages cross-episodic experiences within Transformers to improve learning and generalization across tasks and data qualities.

Findings

CEC improves policy performance in multi-task reinforcement learning.

CEC enhances generalization in imitation learning with mixed-quality data.

Code implementation is publicly available for further research.

Abstract

We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the learning efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer's context, which forms the basis of a curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learning progression and proficiency increase across episodes. Such synergy combined with the potent pattern recognition capabilities of Transformer models delivers a powerful cross-episodic attention mechanism. The effectiveness of CEC is demonstrated under two representative scenarios: one involving multi-task reinforcement learning with discrete control, such as in DeepMind Lab, where the curriculum captures the learning progression in both individual and progressively complex settings; and the other involving imitation learning with mixed-quality data for continuous control, as seen in RoboMimic, where the curriculum captures the improvement in demonstrators' expertise. In all instances, policies resulting from CEC exhibit superior performance and strong generalization. Code is open-sourced at https://cec-agent.github.io/ to facilitate research on Transformer agent learning.