A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning

TL;DR

This paper explores overfitting in continuous deep reinforcement learning, proposing new diagnostic tools and strategies to improve generalization and reduce overfitting risks in complex environments.

Contribution

It introduces novel perspectives on diagnosing and preventing overfitting in continuous RL, emphasizing training diversity and practical insights for researchers.

Findings

Overfitting can be diagnosed in MDPs using new criteria.

Increasing training diversity reduces overfitting risks.

Deep RL methods are inherently brittle without proper regularization.

Abstract

The risks and perils of overfitting in machine learning are well known. However most of the treatment of this, including diagnostic tools and remedies, was developed for the supervised learning case. In this work, we aim to offer new perspectives on the characterization and prevention of overfitting in deep Reinforcement Learning (RL) methods, with a particular focus on continuous domains. We examine several aspects, such as how to define and diagnose overfitting in MDPs, and how to reduce risks by injecting sufficient training diversity. This work complements recent findings on the brittleness of deep RL methods and offers practical observations for RL researchers and practitioners.