On the Generalization of Representations in Reinforcement Learning

TL;DR

This paper provides a theoretical bound on how well state representations in reinforcement learning generalize, based on the concept of effective dimension, and empirically verifies these insights across various methods and environments.

Contribution

It introduces an effective dimension-based bound on generalization error for any state representation in reinforcement learning, bridging theory and empirical analysis.

Findings

Effective dimension explains the generalization behavior of learned representations.

Theoretical bounds align with empirical observations in classic and benchmark environments.

Representations with lower effective dimension tend to generalize better.

Abstract

In reinforcement learning, state representations are used to tractably deal with large problem spaces. State representations serve both to approximate the value function with few parameters, but also to generalize to newly encountered states. Their features may be learned implicitly (as part of a neural network) or explicitly (for example, the successor representation of \citet{dayan1993improving}). While the approximation properties of representations are reasonably well-understood, a precise characterization of how and when these representations generalize is lacking. In this work, we address this gap and provide an informative bound on the generalization error arising from a specific state representation. This bound is based on the notion of effective dimension which measures the degree to which knowing the value at one state informs the value at other states. Our bound applies to any state representation and quantifies the natural tension between representations that generalize well and those that approximate well. We complement our theoretical results with an empirical survey of classic representation learning methods from the literature and results on the Arcade Learning Environment, and find that the generalization behaviour of learned representations is well-explained by their effective dimension.