On the importance of data collection for training general goal-reaching policies

TL;DR

This paper emphasizes the critical role of extensive, high-quality data collection for training versatile goal-reaching policies in reinforcement learning, introducing a new exploration method and analyzing data effects.

Contribution

It introduces ChronoGEM, a principled unsupervised exploration method for uniform state coverage, and demonstrates the importance of data quantity and quality for training general controllers.

Findings

Large data quantities improve policy performance.

High-quality data enhances goal achievement accuracy.

Unsupervised exploration effectively covers state manifolds.

Abstract

Recent advances in ML suggest that the quantity of data available to a model is one of the primary bottlenecks to high performance. Although for language-based tasks there exist almost unlimited amounts of reasonably coherent data to train from, this is generally not the case for Reinforcement Learning, especially when dealing with a novel environment. In effect, even a relatively trivial continuous environment has an almost limitless number of states, but simply sampling random states and actions will likely not provide transitions that are interesting or useful for any potential downstream task. How should one generate massive amounts of useful data given only an MDP with no indication of downstream tasks? Are the quantity and quality of data truly transformative to the performance of a general controller? We propose to answer both of these questions. First, we introduce a principled unsupervised exploration method, ChronoGEM, which aims to achieve uniform coverage over the manifold of achievable states, which we believe is the most reasonable goal given no prior task information. Secondly, we investigate the effects of both data quantity and data quality on the training of a downstream goal-achievement policy, and show that both large quantities and high-quality of data are essential to train a general controller: a high-precision pose-achievement policy capable of attaining a large number of poses over numerous continuous control embodiments including humanoid.