Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning

TL;DR

This paper introduces a pessimism-based framework for zero-shot transfer reinforcement learning that guarantees performance bounds, ensures safety, and mitigates negative transfer when leveraging multiple source domains.

Contribution

It proposes a novel conservative estimation approach that provides performance guarantees and improves transfer safety in reinforcement learning.

Findings

Provides a lower bound on target performance

Ensures monotonic improvement with source domain quality

Develops algorithms with convergence guarantees

Abstract

Transfer reinforcement learning aims to derive a near-optimal policy for a target environment with limited data by leveraging abundant data from related source domains. However, it faces two key challenges: the lack of performance guarantees for the transferred policy, which can lead to undesired actions, and the risk of negative transfer when multiple source domains are involved. We propose a novel framework based on the pessimism principle, which constructs and optimizes a conservative estimation of the target domain's performance. Our framework effectively addresses the two challenges by providing an optimized lower bound on target performance, ensuring safe and reliable decisions, and by exhibiting monotonic improvement with respect to the quality of the source domains, thereby avoiding negative transfer. We construct two types of conservative estimations, rigorously characterize their effectiveness, and develop efficient distributed algorithms with convergence guarantees. Our framework provides a theoretically sound and practically robust solution for transfer learning in reinforcement learning.