Periodic Activation Functions Induce Stationarity

TL;DR

This paper demonstrates that using periodic activation functions in Bayesian neural networks induces stationary Gaussian process priors, improving out-of-domain detection while maintaining in-domain performance.

Contribution

It establishes a theoretical connection between periodic activations and stationary Gaussian process priors, extending beyond sinusoidal functions to other periodic activations.

Findings

Periodic activations induce stationary Gaussian process priors.

Models with periodic activations perform well on in-domain data.

Periodic activations improve out-of-domain detection sensitivity.

Abstract

Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that `know what they do not know' by introducing inductive biases in the function space. We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. Furthermore, we show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.