EPOpt: Learning Robust Neural Network Policies Using Model Ensembles

TL;DR

EPOpt introduces an ensemble-based reinforcement learning method that enhances policy robustness and generalization across diverse and unmodeled real-world domains through adversarial training and adaptive source domain weighting.

Contribution

The paper presents EPOpt, a novel algorithm combining model ensembles, adversarial training, and Bayesian adaptation to improve policy robustness and domain generalization in reinforcement learning.

Findings

EPOpt achieves robust policies across varied simulated domains.

The adaptive source domain weighting improves transfer to real-world tasks.

EPOpt outperforms traditional methods in domain generalization experiments.

Abstract

Sample complexity and safety are major challenges when learning policies with reinforcement learning for real-world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model-based methods where the real-world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.