Learning Versatile Skills with Curriculum Masking

TL;DR

CurrMask is a curriculum-based masking pretraining method that dynamically adjusts masking schemes during offline RL training, enabling the model to learn versatile skills of varying complexity and perform well on multiple downstream tasks.

Contribution

We introduce CurrMask, a novel curriculum masking approach that improves skill learning in offline RL by dynamically adjusting masking schemes during pretraining.

Findings

Superior zero-shot performance on skill prompting tasks

Effective goal-conditioned planning results

Competitive finetuning performance on offline RL tasks

Abstract

Masked prediction has emerged as a promising pretraining paradigm in offline reinforcement learning (RL) due to its versatile masking schemes, enabling flexible inference across various downstream tasks with a unified model. Despite the versatility of masked prediction, it remains unclear how to balance the learning of skills at different levels of complexity. To address this, we propose CurrMask, a curriculum masking pretraining paradigm for sequential decision making. Motivated by how humans learn by organizing knowledge in a curriculum, CurrMask adjusts its masking scheme during pretraining for learning versatile skills. Through extensive experiments, we show that CurrMask exhibits superior zero-shot performance on skill prompting tasks, goal-conditioned planning tasks, and competitive finetuning performance on offline RL tasks. Additionally, our analysis of training dynamics reveals that CurrMask gradually acquires skills of varying complexity by dynamically adjusting its masking scheme.