# Measuring and Characterizing Generalization in Deep Reinforcement Learning

## TL;DR

This paper critically examines how well deep reinforcement learning agents truly generalize, proposing new definitions and evaluation methods, and revealing that current agents often fail to generalize to slightly different states.

## Contribution

It introduces practical evaluation techniques for generalization in RL and challenges assumptions about deep RL agents' ability to learn generalized representations.

## Key findings

- Deep RL agents perform poorly on states similar to on-policy states.
- Current deep Q-networks lack robust generalization capabilities.
- More analysis is needed to confirm if deep RL truly learns generalized representations.

## Abstract

Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions based on an agent's performance in on-policy, off-policy, and unreachable states. We propose a set of practical methods for evaluating agents with these definitions of generalization. We demonstrate these techniques on a common benchmark task for deep RL, and we show that the learned networks make poor decisions for states that differ only slightly from on-policy states, even though those states are not selected adversarially. Taken together, these results call into question the extent to which deep Q-networks learn generalized representations, and suggest that more experimentation and analysis is necessary before claims of representation learning can be supported.