Neural Networks Fail to Learn Periodic Functions and How to Fix It

TL;DR

This paper investigates why neural networks struggle with learning periodic functions, proves the failure of standard activations in extrapolation, and proposes a new activation function with a periodic bias that improves learning in real-world data.

Contribution

It introduces a novel activation function, $x + \\sin^2(x)$, that embeds a periodic inductive bias into neural networks, enabling better learning of periodic functions.

Findings

Standard activations fail to extrapolate periodic functions.

The proposed activation improves periodic function learning.

Enhanced performance on temperature and financial data prediction.

Abstract

Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.