Understanding the World Through Action

TL;DR

This paper proposes a reinforcement learning framework that leverages large unlabeled datasets with self-supervised objectives, aiming to improve the scalability and alignment of machine learning models with downstream tasks.

Contribution

It introduces a novel reinforcement learning-based approach for utilizing unlabeled data through self-supervised objectives combined with offline RL techniques.

Findings

Framework aligns better with downstream tasks

Leverages large unlabeled datasets effectively

Builds on recent advances in self-supervised RL

Abstract

The recent history of machine learning research has taught us that machine learning methods can be most effective when they are provided with very large, high-capacity models, and trained on very large and diverse datasets. This has spurred the community to search for ways to remove any bottlenecks to scale. Often the foremost among such bottlenecks is the need for human effort, including the effort of curating and labeling datasets. As a result, considerable attention in recent years has been devoted to utilizing unlabeled data, which can be collected in vast quantities. However, some of the most widely used methods for training on such unlabeled data themselves require human-designed objective functions that must correlate in some meaningful way to downstream tasks. I will argue that a general, principled, and powerful framework for utilizing unlabeled data can be derived from reinforcement learning, using general purpose unsupervised or self-supervised reinforcement learning objectives in concert with offline reinforcement learning methods that can leverage large datasets. I will discuss how such a procedure is more closely aligned with potential downstream tasks, and how it could build on existing techniques that have been developed in recent years.