Adaptive Policy Transfer in Reinforcement Learning

TL;DR

This paper introduces a new method for reinforcement learning that combines adaptation and exploration to improve policy transfer efficiency across related tasks, inspired by biological skill transfer.

Contribution

The paper proposes a novel 'Adapt-to-Learn' mechanism that enhances policy transfer by seamlessly integrating adaptation and exploration, reducing sample complexity.

Findings

The method achieves significantly reduced sample complexity in transferring skills.

It enables seamless combination of adaptation and exploration during policy transfer.

The approach demonstrates robustness in transferring policies between related tasks.

Abstract

Efficient and robust policy transfer remains a key challenge for reinforcement learning to become viable for real-wold robotics. Policy transfer through warm initialization, imitation, or interacting over a large set of agents with randomized instances, have been commonly applied to solve a variety of Reinforcement Learning tasks. However, this seems far from how skill transfer happens in the biological world: Humans and animals are able to quickly adapt the learned behaviors between similar tasks and learn new skills when presented with new situations. Here we seek to answer the question: Will learning to combine adaptation and exploration lead to a more efficient transfer of policies between domains? We introduce a principled mechanism that can "Adapt-to-Learn", that is adapt the source policy to learn to solve a target task with significant transition differences and uncertainties. We show that the presented method learns to seamlessly combine learning from adaptation and exploration and leads to a robust policy transfer algorithm with significantly reduced sample complexity in transferring skills between related tasks.