The Curse of Diversity in Ensemble-Based Exploration

TL;DR

Training diverse ensembles in deep reinforcement learning can unexpectedly harm individual performance due to data sharing issues, but representation learning methods like CERL can help mitigate this problem.

Contribution

This paper identifies the 'curse of diversity' in ensemble-based exploration and proposes a novel representation learning approach, CERL, to address it.

Findings

Ensemble diversity can impair individual agent performance.

Larger replay buffers or smaller ensembles do not reliably solve the issue.

Representation learning via CERL effectively counters the curse.

Abstract

We uncover a surprising phenomenon in deep reinforcement learning: training a diverse ensemble of data-sharing agents -- a well-established exploration strategy -- can significantly impair the performance of the individual ensemble members when compared to standard single-agent training. Through careful analysis, we attribute the degradation in performance to the low proportion of self-generated data in the shared training data for each ensemble member, as well as the inefficiency of the individual ensemble members to learn from such highly off-policy data. We thus name this phenomenon the curse of diversity. We find that several intuitive solutions -- such as a larger replay buffer or a smaller ensemble size -- either fail to consistently mitigate the performance loss or undermine the advantages of ensembling. Finally, we demonstrate the potential of representation learning to counteract the curse of diversity with a novel method named Cross-Ensemble Representation Learning (CERL) in both discrete and continuous control domains. Our work offers valuable insights into an unexpected pitfall in ensemble-based exploration and raises important caveats for future applications of similar approaches.