Offline Reinforcement Learning with Closed-Form Policy Improvement Operators

TL;DR

This paper introduces closed-form policy improvement operators for offline reinforcement learning, leveraging behavior constraints and Gaussian mixture models to enhance policy optimization, demonstrated through superior performance on standard benchmarks.

Contribution

The paper proposes novel closed-form policy improvement operators using first-order Taylor approximation and Gaussian mixture models for behavior policies in offline RL.

Findings

Outperforms state-of-the-art algorithms on D4RL benchmarks

Introduces a linear approximation of the policy objective

Provides a closed-form solution for policy improvement

Abstract

Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. By exploiting historical transitions, a policy is trained to maximize a learned value function while constrained by the behavior policy to avoid a significant distributional shift. In this paper, we propose our closed-form policy improvement operators. We make a novel observation that the behavior constraint naturally motivates the use of first-order Taylor approximation, leading to a linear approximation of the policy objective. Additionally, as practical datasets are usually collected by heterogeneous policies, we model the behavior policies as a Gaussian Mixture and overcome the induced optimization difficulties by leveraging the LogSumExp's lower bound and Jensen's Inequality, giving rise to a closed-form policy improvement operator. We instantiate offline RL algorithms with our novel policy improvement operators and empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark. Our code is available at https://cfpi-icml23.github.io/.