Positional Embedding-Aware Activations

TL;DR

This paper introduces SPDER, a neural network architecture with a novel activation function that learns positional embeddings, significantly improving training speed and representation quality for images and audio without hyperparameter tuning.

Contribution

The paper proposes a simple MLP with a sinusoidal-damped activation function that naturally learns positional embeddings and overcomes spectral bias, achieving state-of-the-art results in image and audio representation.

Findings

Speeds up training by 10x

Achieves 1,500-50,000x lower loss than previous methods in image representation

Excels in downstream tasks like super-resolution and video interpolation

Abstract

We present a neural network architecture designed to naturally learn a positional embedding and overcome the spectral bias towards lower frequencies faced by conventional activation functions. Our proposed architecture, SPDER, is a simple MLP that uses an activation function composed of a sinusoidal multiplied by a sublinear function, called the damping function. The sinusoidal enables the network to automatically learn the positional embedding of an input coordinate while the damping passes on the actual coordinate value by preventing it from being projected down to within a finite range of values. Our results indicate that SPDERs speed up training by 10x and converge to losses 1,500-50,000x lower than that of the state-of-the-art for image representation. SPDER is also state-of-the-art in audio representation. The superior representation capability allows SPDER to also excel on multiple downstream tasks such as image super-resolution and video frame interpolation. We provide intuition as to why SPDER significantly improves fitting compared to that of other INR methods while requiring no hyperparameter tuning or preprocessing.