A Survey Analyzing Generalization in Deep Reinforcement Learning

TL;DR

This survey comprehensively analyzes the challenges and solutions related to the generalization capabilities of deep reinforcement learning policies, highlighting fundamental issues and categorizing various approaches to improve robustness.

Contribution

It provides a formal analysis of generalization in deep reinforcement learning and categorizes solution strategies to address overfitting and enhance policy robustness.

Findings

Identifies overfitting as a key challenge in deep reinforcement learning

Categorizes solution approaches including regularization and adversarial analysis

Offers a broad overview of current advancements and future directions

Abstract

Reinforcement learning research obtained significant success and attention with the utilization of deep neural networks to solve problems in high dimensional state or action spaces. While deep reinforcement learning policies are currently being deployed in many different fields from medical applications to large language models, there are still ongoing questions the field is trying to answer on the generalization capabilities of deep reinforcement learning policies. In this paper, we will formalize and analyze generalization in deep reinforcement learning. We will explain the fundamental reasons why deep reinforcement learning policies encounter overfitting problems that limit their generalization capabilities. Furthermore, we will categorize and explain the manifold solution approaches to increase generalization, and overcome overfitting in deep reinforcement learning policies. From exploration to adversarial analysis and from regularization to robustness our paper provides an analysis on a wide range of subfields within deep reinforcement learning with a broad scope and in-depth view. We believe our study can provide a compact guideline for the current advancements in deep reinforcement learning, and help to construct robust deep neural policies with higher generalization skills.