Renaissance Robot: Optimal Transport Policy Fusion for Learning Diverse Skills

TL;DR

This paper introduces a post-hoc policy fusion method using Optimal Transport to combine knowledge from multiple RL agents, enabling faster learning of new skills in robotics.

Contribution

It proposes a novel Optimal Transport-based policy fusion technique that improves initialization for new tasks, reducing training time and computational resources.

Findings

Unified policies enable quicker skill acquisition.

Significant reduction in training time for new tasks.

Effective knowledge consolidation across diverse agents.

Abstract

Deep reinforcement learning (RL) is a promising approach to solving complex robotics problems. However, the process of learning through trial-and-error interactions is often highly time-consuming, despite recent advancements in RL algorithms. Additionally, the success of RL is critically dependent on how well the reward-shaping function suits the task, which is also time-consuming to design. As agents trained on a variety of robotics problems continue to proliferate, the ability to reuse their valuable learning for new domains becomes increasingly significant. In this paper, we propose a post-hoc technique for policy fusion using Optimal Transport theory as a robust means of consolidating the knowledge of multiple agents that have been trained on distinct scenarios. We further demonstrate that this provides an improved weights initialisation of the neural network policy for learning new tasks, requiring less time and computational resources than either retraining the parent policies or training a new policy from scratch. Ultimately, our results on diverse agents commonly used in deep RL show that specialised knowledge can be unified into a "Renaissance agent", allowing for quicker learning of new skills.