Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimization

TL;DR

This paper introduces a novel value enhancement method for offline reinforcement learning that improves policy performance and convergence rates, especially in data-limited high-stakes domains, using a generalizable approach applicable to neural network policies.

Contribution

The paper proposes a new value enhancement technique for offline RL that guarantees non-worse policies and accelerates convergence to optimality under mild conditions.

Findings

Method improves policy value compared to initial policies.

Accelerates convergence to optimal policy under mild conditions.

Demonstrates superior performance in extensive numerical studies.

Abstract

Reinforcement learning (RL) is a powerful machine learning technique that enables an intelligent agent to learn an optimal policy that maximizes the cumulative rewards in sequential decision making. Most of methods in the existing literature are developed in \textit{online} settings where the data are easy to collect or simulate. Motivated by high stake domains such as mobile health studies with limited and pre-collected data, in this paper, we study \textit{offline} reinforcement learning methods. To efficiently use these datasets for policy optimization, we propose a novel value enhancement method to improve the performance of a given initial policy computed by existing state-of-the-art RL algorithms. Specifically, when the initial policy is not consistent, our method will output a policy whose value is no worse and often better than that of the initial policy. When the initial policy is consistent, under some mild conditions, our method will yield a policy whose value converges to the optimal one at a faster rate than the initial policy, achieving the desired ``value enhancement" property. The proposed method is generally applicable to any parametrized policy that belongs to certain pre-specified function class (e.g., deep neural networks). Extensive numerical studies are conducted to demonstrate the superior performance of our method.