Periodic Intra-Ensemble Knowledge Distillation for Reinforcement Learning

TL;DR

This paper introduces PIEKD, a framework that enhances reinforcement learning by periodically sharing knowledge within an ensemble of policies, leading to improved sample efficiency on benchmark tasks.

Contribution

The paper proposes a novel periodic intra-ensemble knowledge distillation method that boosts RL performance by combining ensemble actions with knowledge sharing.

Findings

PIEKD outperforms state-of-the-art RL methods in sample efficiency.

Periodic knowledge sharing improves policy learning.

Ablation studies confirm the effectiveness of PIEKD.

Abstract

Off-policy ensemble reinforcement learning (RL) methods have demonstrated impressive results across a range of RL benchmark tasks. Recent works suggest that directly imitating experts' policies in a supervised manner before or during the course of training enables faster policy improvement for an RL agent. Motivated by these recent insights, we propose Periodic Intra-Ensemble Knowledge Distillation (PIEKD). PIEKD is a learning framework that uses an ensemble of policies to act in the environment while periodically sharing knowledge amongst policies in the ensemble through knowledge distillation. Our experiments demonstrate that PIEKD improves upon a state-of-the-art RL method in sample efficiency on several challenging MuJoCo benchmark tasks. Additionally, we perform ablation studies to better understand PIEKD.