From Taylor Series to Fourier Synthesis: The Periodic Linear Unit

TL;DR

This paper introduces the Periodic Linear Unit (PLU), a sine-wave based activation function that enhances neural network expressiveness and efficiency, enabling simple models to solve complex tasks like spiral classification.

Contribution

The paper proposes the PLU activation and Repulsive Reparameterization, enabling neural networks to act as Fourier synthesizers rather than Taylor approximators, improving efficiency and expressiveness.

Findings

A minimal MLP with two PLU neurons solves spiral classification.

PLU outperforms standard activations in expressive power.

Demonstrates exponential parameter efficiency gains.

Abstract

The dominant paradigm in modern neural networks relies on simple, monotonically-increasing activation functions like ReLU. While effective, this paradigm necessitates large, massively-parameterized models to approximate complex functions. In this paper, we introduce the Periodic Linear Unit (PLU), a learnable sine-wave based activation with periodic non-monotonicity. PLU is designed for maximum expressive power and numerical stability, achieved through its formulation and a paired innovation we term Repulsive Reparameterization, which prevents the activation from collapsing into a non-expressive linear function. We demonstrate that a minimal MLP with only two PLU neurons can solve the spiral classification task, a feat impossible for equivalent networks using standard activations. This suggests a paradigm shift from networks as piecewise Taylor-like approximators to powerful Fourier-like function synthesizers, achieving exponential gains in parameter efficiency by placing intelligence in the neuron itself.