Offline Reinforcement Learning with Behavioral Supervisor Tuning

TL;DR

This paper introduces TD3-BST, an offline RL algorithm that uses behavioral supervisor tuning to improve policy learning without extensive hyperparameter tuning, making it more practical for real-world applications.

Contribution

The paper proposes TD3-BST, a novel offline RL method that trains an uncertainty model to guide policy actions within dataset support, reducing tuning requirements.

Findings

TD3-BST outperforms previous offline RL methods on benchmarks.

It achieves state-of-the-art results without per-dataset hyperparameter tuning.

The approach simplifies offline RL deployment in practical settings.

Abstract

Offline reinforcement learning (RL) algorithms are applied to learn performant, well-generalizing policies when provided with a static dataset of interactions. Many recent approaches to offline RL have seen substantial success, but with one key caveat: they demand substantial per-dataset hyperparameter tuning to achieve reported performance, which requires policy rollouts in the environment to evaluate; this can rapidly become cumbersome. Furthermore, substantial tuning requirements can hamper the adoption of these algorithms in practical domains. In this paper, we present TD3 with Behavioral Supervisor Tuning (TD3-BST), an algorithm that trains an uncertainty model and uses it to guide the policy to select actions within the dataset support. TD3-BST can learn more effective policies from offline datasets compared to previous methods and achieves the best performance across challenging benchmarks without requiring per-dataset tuning.