A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning

TL;DR

This paper introduces a non-monolithic exploration method for offline-to-online reinforcement learning that balances offline policy exploitation with online policy exploration, improving upon existing approaches like Policy Expansion (PEX).

Contribution

The proposed method effectively harmonizes offline exploitation and online exploration without modifying the offline policy, leading to better performance than PEX.

Findings

Outperforms PEX in downstream tasks

Balances exploitation and exploration effectively

Enhances data efficiency in RL

Abstract

Offline-to-online reinforcement learning (RL) leverages both pre-trained offline policies and online policies trained for downstream tasks, aiming to improve data efficiency and accelerate performance enhancement. An existing approach, Policy Expansion (PEX), utilizes a policy set composed of both policies without modifying the offline policy for exploration and learning. However, this approach fails to ensure sufficient learning of the online policy due to an excessive focus on exploration with both policies. Since the pre-trained offline policy can assist the online policy in exploiting a downstream task based on its prior experience, it should be executed effectively and tailored to the specific requirements of the downstream task. In contrast, the online policy, with its immature behavioral strategy, has the potential for exploration during the training phase. Therefore, our research focuses on harmonizing the advantages of the offline policy, termed exploitation, with those of the online policy, referred to as exploration, without modifying the offline policy. In this study, we propose an innovative offline-to-online RL method that employs a non-monolithic exploration approach. Our methodology demonstrates superior performance compared to PEX.