Measuring Learning Progress via Gradient-Momentum Coupling

TL;DR

This paper introduces Gradient-Momentum Coupling (GMC), a novel signal derived from optimization dynamics that better measures meaningful learning progress in reinforcement learning by filtering noise and prioritizing useful samples.

Contribution

The paper proposes GMC, a new metric based on gradient and momentum coupling, which improves noise robustness and learning efficiency over traditional signals like prediction error.

Findings

GMC effectively filters noise and oscillations in learning signals.

Replacing prediction error with GMC enhances robustness to observation noise.

GMC leads to emergent curriculum learning by prioritizing tasks based on learning speed.

Abstract

Measuring learning progress is essential for curiosity-driven exploration in reinforcement learning, but widely used signals such as prediction error often fail to distinguish meaningful, learnable patterns from random noise. This paper proposes Gradient-Momentum Coupling (GMC), a signal derived from optimization dynamics that quantifies how useful each sample's gradient is for ongoing learning by measuring its per-parameter normalized absolute product with the momentum from previous gradients. By leveraging momentum's natural filtering of noise and oscillations, GMC identifies samples that contribute to ongoing parameter updates. Controlled experiments demonstrate noise robustness and emergent curriculum learning, with the signal prioritizing tasks by learning speed rather than difficulty. Experiments on MiniGrid suggest that replacing prediction error with GMC within existing curiosity-driven architectures can improve robustness to observation noise.