Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning

TL;DR

This paper introduces UWAC, an offline RL algorithm that effectively manages uncertainty by down-weighting OOD actions, leading to improved stability and performance on various tasks.

Contribution

The paper proposes a novel uncertainty-aware actor-critic method using dropout-based estimation to enhance offline RL stability and performance.

Findings

UWAC improves training stability.

UWAC outperforms existing offline RL methods.

Significant gains on datasets with sparse human demonstrations.

Abstract

Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We hypothesize that a key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly. Implementation-wise, we adopt a practical and effective dropout-based uncertainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-art baseline on datasets with sparse demonstrations collected from human experts.