Multi-Environment Pretraining Enables Transfer to Action Limited Datasets

TL;DR

This paper introduces ALPT, a pretraining method that uses inverse dynamics modeling to generate action labels in unannotated datasets, enabling effective transfer learning in environments with limited action data.

Contribution

The paper presents a novel pretraining approach combining multi-environment data and inverse dynamics modeling to improve reinforcement learning in action-sparse settings.

Findings

Significant performance improvements in game environments with minimal annotated data.

Effective action label generation even across environments with no shared actions.

Enhanced generalization capabilities demonstrated on benchmark tasks.

Abstract

Using massive datasets to train large-scale models has emerged as a dominant approach for broad generalization in natural language and vision applications. In reinforcement learning, however, a key challenge is that available data of sequential decision making is often not annotated with actions - for example, videos of game-play are much more available than sequences of frames paired with their logged game controls. We propose to circumvent this challenge by combining large but sparsely-annotated datasets from a \emph{target} environment of interest with fully-annotated datasets from various other \emph{source} environments. Our method, Action Limited PreTraining (ALPT), leverages the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data in the target environment. We show that utilizing even one additional environment dataset of labelled data during IDM pretraining gives rise to substantial improvements in generating action labels for unannotated sequences. We evaluate our method on benchmark game-playing environments and show that we can significantly improve game performance and generalization capability compared to other approaches, using annotated datasets equivalent to only $12$ minutes of gameplay. Highlighting the power of IDM, we show that these benefits remain even when target and source environments share no common actions.