Investigating Generalisation in Continuous Deep Reinforcement Learning

TL;DR

This paper examines the generalisation challenges in deep reinforcement learning, highlighting the gap between training performance and real-world robustness, and evaluates techniques to improve generalisation under domain shifts.

Contribution

It characterises sources of uncertainty affecting generalisation, introduces a new benchmark, and empirically evaluates state-of-the-art methods for robustness in Deep RL.

Findings

Training performance alone is misleading for generalisation.

Common techniques vary in robustness under domain shift.

Certain methods significantly improve generalisation in Deep RL.

Abstract

Deep Reinforcement Learning has shown great success in a variety of control tasks. However, it is unclear how close we are to the vision of putting Deep RL into practice to solve real world problems. In particular, common practice in the field is to train policies on largely deterministic simulators and to evaluate algorithms through training performance alone, without a train/test distinction to ensure models generalise and are not overfitted. Moreover, it is not standard practice to check for generalisation under domain shift, although robustness to such system change between training and testing would be necessary for real-world Deep RL control, for example, in robotics. In this paper we study these issues by first characterising the sources of uncertainty that provide generalisation challenges in Deep RL. We then provide a new benchmark and thorough empirical evaluation of generalisation challenges for state of the art Deep RL methods. In particular, we show that, if generalisation is the goal, then common practice of evaluating algorithms based on their training performance leads to the wrong conclusions about algorithm choice. Finally, we evaluate several techniques for improving generalisation and draw conclusions about the most robust techniques to date.