Implicit Offline Reinforcement Learning via Supervised Learning

TL;DR

This paper introduces an implicit supervised learning approach for offline reinforcement learning that leverages return information, outperforming explicit models in robotic skill acquisition from fixed datasets.

Contribution

It extends offline RL via supervised learning to implicit models, demonstrating their effectiveness and unifying them with existing algorithms.

Findings

Implicit models outperform explicit ones in robotic tasks.

The method is effective on high-dimensional manipulation and locomotion tasks.

A unified framework connects implicit and explicit RL via supervised learning.

Abstract

Offline Reinforcement Learning (RL) via Supervised Learning is a simple and effective way to learn robotic skills from a dataset collected by policies of different expertise levels. It is as simple as supervised learning and Behavior Cloning (BC), but takes advantage of return information. On datasets collected by policies of similar expertise, implicit BC has been shown to match or outperform explicit BC. Despite the benefits of using implicit models to learn robotic skills via BC, offline RL via Supervised Learning algorithms have been limited to explicit models. We show how implicit models can leverage return information and match or outperform explicit algorithms to acquire robotic skills from fixed datasets. Furthermore, we show the close relationship between our implicit methods and other popular RL via Supervised Learning algorithms to provide a unified framework. Finally, we demonstrate the effectiveness of our method on high-dimension manipulation and locomotion tasks.