Semi-pessimistic Reinforcement Learning

TL;DR

This paper introduces a semi-pessimistic reinforcement learning approach that leverages large amounts of unlabeled data to improve policy learning in offline RL, addressing distributional shift and reward scarcity.

Contribution

The paper proposes a novel semi-pessimistic RL method that simplifies reward estimation, is flexible across algorithms, and guarantees improvement with abundant unlabeled data.

Findings

Method outperforms existing solutions analytically and numerically.

Applicable to both model-free and model-based RL algorithms.

Demonstrated effectiveness in a Parkinson's disease application.

Abstract

Offline reinforcement learning (RL) aims to learn an optimal policy from pre-collected data. However, it faces challenges of distributional shift, where the learned policy may encounter unseen scenarios not covered in the offline data. Additionally, numerous applications suffer from a scarcity of labeled reward data. Relying on labeled data alone often leads to a narrow state-action distribution, further amplifying the distributional shift, and resulting in suboptimal policy learning. To address these issues, we first recognize that the volume of unlabeled data is typically substantially larger than that of labeled data. We then propose a semi-pessimistic RL method to effectively leverage abundant unlabeled data. Our approach offers several advantages. It considerably simplifies the learning process, as it seeks a lower bound of the reward function, rather than that of the Q-function or state transition function. It is highly flexible, and can be integrated with a range of model-free and model-based RL algorithms. It enjoys the guaranteed improvement when utilizing vast unlabeled data, but requires much less restrictive conditions. We compare our method with a number of alternative solutions, both analytically and numerically, and demonstrate its clear competitiveness. We further illustrate with an application to adaptive deep brain stimulation for Parkinson's disease.