CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization

TL;DR

This paper introduces CAWR, a robust offline RL method that mitigates over-conservatism caused by poor exploration data by using robust loss functions and prioritized experience replay, leading to improved policy learning.

Contribution

The paper proposes CAWR, a novel offline RL algorithm that addresses over-conservatism from data corruption through robust loss functions and experience replay filtering.

Findings

CAWR outperforms existing methods on D4RL benchmarks.

It effectively filters poor exploration data during training.

The approach significantly improves policy quality from suboptimal datasets.

Abstract

Offline reinforcement learning (offline RL) algorithms often require additional constraints or penalty terms to address distribution shift issues, such as adding implicit or explicit policy constraints during policy optimization to reduce the estimation bias of functions. This paper focuses on a limitation of the Advantage-Weighted Regression family (AWRs), i.e., the potential for learning over-conservative policies due to data corruption, specifically the poor explorations in suboptimal offline data. We study it from two perspectives: (1) how poor explorations impact the theoretically optimal policy based on KL divergence, and (2) how such poor explorations affect the approximation of the theoretically optimal policy. We prove that such over-conservatism is mainly caused by the sensitivity of the loss function for policy optimization to poor explorations, and the proportion of poor explorations in offline datasets. To address this concern, we propose Corruption-Averse Advantage-Weighted Regression (CAWR), which incorporates a set of robust loss functions during policy optimization and an advantage-based prioritized experience replay method to filter out poor explorations. Numerical experiments on the D4RL benchmark show that our method can learn superior policies from suboptimal offline data, significantly enhancing the performance of policy optimization.