Understanding the Evolution of Linear Regions in Deep Reinforcement Learning

TL;DR

This paper investigates how the complexity of deep reinforcement learning policies, measured by linear regions induced by ReLU networks, evolves during training across various tasks, revealing moderate increases in region density and insights into policy complexity.

Contribution

It provides empirical analysis of linear region evolution in deep RL policies, comparing results with supervised learning theories and highlighting that policy complexity does not significantly grow during training.

Findings

Region density increases only moderately during training.

Trajectory lengths increase, causing perceived decreases in region density.

Policy complexity does not mainly stem from growth in linear regions.

Abstract

Policies produced by deep reinforcement learning are typically characterised by their learning curves, but they remain poorly understood in many other respects. ReLU-based policies result in a partitioning of the input space into piecewise linear regions. We seek to understand how observed region counts and their densities evolve during deep reinforcement learning using empirical results that span a range of continuous control tasks and policy network dimensions. Intuitively, we may expect that during training, the region density increases in the areas that are frequently visited by the policy, thereby affording fine-grained control. We use recent theoretical and empirical results for the linear regions induced by neural networks in supervised learning settings for grounding and comparison of our results. Empirically, we find that the region density increases only moderately throughout training, as measured along fixed trajectories coming from the final policy. However, the trajectories themselves also increase in length during training, and thus the region densities decrease as seen from the perspective of the current trajectory. Our findings suggest that the complexity of deep reinforcement learning policies does not principally emerge from a significant growth in the complexity of functions observed on-and-around trajectories of the policy.