Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

TL;DR

This paper introduces P2PDRL, a peer-to-peer distillation method for reinforcement learning that improves robustness and generalization across diverse randomized environments by mutual knowledge exchange among multiple agents.

Contribution

The paper proposes a novel peer-to-peer distillation approach for reinforcement learning that enhances robustness and generalization in domain-randomized settings.

Findings

P2PDRL outperforms baseline methods in continuous control tasks.

P2PDRL achieves better robustness across wider domain randomizations.

P2PDRL improves generalization to unseen environments.

Abstract

In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback-Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.