Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learning

TL;DR

This paper investigates the role of periodic activation functions in reinforcement learning, revealing they tend to learn high frequencies, improve sample efficiency but may overfit, with regularization helping balance these effects.

Contribution

It provides empirical analysis of periodic activations in RL, clarifying their frequency learning behavior and impact on generalization, and suggests regularization as a mitigation strategy.

Findings

Periodic activations converge to high frequencies regardless of initialization.

They improve sample efficiency but worsen generalization with noisy observations.

Weight decay regularization partially mitigates overfitting, balancing learning and generalization.

Abstract

Periodic activation functions, often referred to as learned Fourier features have been widely demonstrated to improve sample efficiency and stability in a variety of deep RL algorithms. Potentially incompatible hypotheses have been made about the source of these improvements. One is that periodic activations learn low frequency representations and as a result avoid overfitting to bootstrapped targets. Another is that periodic activations learn high frequency representations that are more expressive, allowing networks to quickly fit complex value functions. We analyse these claims empirically, finding that periodic representations consistently converge to high frequencies regardless of their initialisation frequency. We also find that while periodic activation functions improve sample efficiency, they exhibit worse generalization on states with added observation noise -- especially when compared to otherwise equivalent networks with ReLU activation functions. Finally, we show that weight decay regularization is able to partially offset the overfitting of periodic activation functions, delivering value functions that learn quickly while also generalizing.