Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning

TL;DR

This paper systematically compares various demonstration-augmented RL methods, finding that simple data reuse and behavior cloning outperform complex offline RL pretraining in improving online sample efficiency.

Contribution

It provides an extensive empirical study classifying and analyzing demonstration-augmented RL approaches, identifying effective hybrid strategies for sample-efficient learning.

Findings

Direct offline data reuse improves sample efficiency.

Behavior cloning initialization outperforms complex offline RL pretraining.

Combining strategies can lead to cumulative benefits.

Abstract

Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.