MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning

TL;DR

The paper introduces MEPG, a minimalist ensemble policy gradient framework for deep reinforcement learning that improves generalization and robustness without additional computational costs by integrating multiple models into a single network using dropout.

Contribution

It proposes a novel ensemble RL framework that maintains ensemble properties with minimal resources through a dropout-based Bellman update, enhancing generalization and robustness.

Findings

MEPG outperforms or matches state-of-the-art ensemble methods in experiments.

It maintains ensemble properties with a single model using dropout.

The framework does not increase computational resource requirements.

Abstract

During the training of a reinforcement learning (RL) agent, the distribution of training data is non-stationary as the agent's behavior changes over time. Therefore, there is a risk that the agent is overspecialized to a particular distribution and its performance suffers in the larger picture. Ensemble RL can mitigate this issue by learning a robust policy. However, it suffers from heavy computational resource consumption due to the newly introduced value and policy functions. In this paper, to avoid the notorious resources consumption issue, we design a novel and simple ensemble deep RL framework that integrates multiple models into a single model. Specifically, we propose the \underline{M}inimalist \underline{E}nsemble \underline{P}olicy \underline{G}radient framework (MEPG), which introduces minimalist ensemble consistent Bellman update by utilizing a modified dropout operator. MEPG holds ensemble property by keeping the dropout consistency of both sides of the Bellman equation. Additionally, the dropout operator also increases MEPG's generalization capability. Moreover, we theoretically show that the policy evaluation phase in the MEPG maintains two synchronized deep Gaussian Processes. To verify the MEPG framework's ability to generalize, we perform experiments on the gym simulator, which presents that the MEPG framework outperforms or achieves a similar level of performance as the current state-of-the-art ensemble methods and model-free methods without increasing additional computational resource costs.