Transfer Reinforcement Learning across Homotopy Classes

TL;DR

This paper addresses the challenge of transfer reinforcement learning across different homotopy classes of tasks, proposing a novel fine-tuning algorithm that improves sample efficiency and transfer success in robotics simulations.

Contribution

The paper introduces Ease-In-Ease-Out fine-tuning, a new algorithm enabling effective transfer across homotopy classes in reinforcement learning, with empirical validation on robotics environments.

Findings

Ease-In-Ease-Out fine-tuning outperforms baselines in sample efficiency.

Fine-tuning across homotopy classes is more challenging and often requires more interactions.

The proposed method enables successful transfer where traditional fine-tuning fails.

Abstract

The ability for robots to transfer their learned knowledge to new tasks -- where data is scarce -- is a fundamental challenge for successful robot learning. While fine-tuning has been well-studied as a simple but effective transfer approach in the context of supervised learning, it is not as well-explored in the context of reinforcement learning. In this work, we study the problem of fine-tuning in transfer reinforcement learning when tasks are parameterized by their reward functions, which are known beforehand. We conjecture that fine-tuning drastically underperforms when source and target trajectories are part of different homotopy classes. We demonstrate that fine-tuning policy parameters across homotopy classes compared to fine-tuning within a homotopy class requires more interaction with the environment, and in certain cases is impossible. We propose a novel fine-tuning algorithm, Ease-In-Ease-Out fine-tuning, that consists of a relaxing stage and a curriculum learning stage to enable transfer learning across homotopy classes. Finally, we evaluate our approach on several robotics-inspired simulated environments and empirically verify that the Ease-In-Ease-Out fine-tuning method can successfully fine-tune in a sample-efficient way compared to existing baselines.