Learning values across many orders of magnitude

TL;DR

The paper introduces an adaptive normalization method for value learning algorithms, enabling them to handle varying value scales over time, which improves generality and removes the need for domain-specific reward clipping.

Contribution

It proposes a novel adaptive normalization technique for value functions that enhances learning stability and removes the reliance on reward clipping in reinforcement learning.

Findings

Improved performance in Atari game benchmarks.

Elimination of reward clipping heuristic.

Enhanced stability in value approximation over time.

Abstract

Most learning algorithms are not invariant to the scale of the function that is being approximated. We propose to adaptively normalize the targets used in learning. This is useful in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the policy of behavior. Our main motivation is prior work on learning to play Atari games, where the rewards were all clipped to a predetermined range. This clipping facilitates learning across many different games with a single learning algorithm, but a clipped reward function can result in qualitatively different behavior. Using the adaptive normalization we can remove this domain-specific heuristic without diminishing overall performance.