Understanding Sinusoidal Neural Networks

TL;DR

This paper analyzes the structure and representation capacity of sinusoidal MLPs, providing theoretical insights into their harmonic properties, compactness, and periodicity, with applications to training and initialization.

Contribution

It offers a harmonic expansion framework for sinusoidal MLPs, proving their frequency composition and control mechanisms, and relates them to Fourier series.

Findings

Sinusoidal neurons expand as harmonic sums with integer linear frequency combinations.

The network's periodicity is determined by input neurons, linking to Fourier series.

Provides bounds and methods for controlling approximation and training of sinusoidal MLPs.

Abstract

In this work, we investigate the structure and representation capacity of sinusoidal MLPs - multilayer perceptron networks that use sine as the activation function. These neural networks (known as neural fields) have become fundamental in representing common signals in computer graphics, such as images, signed distance functions, and radiance fields. This success can be primarily attributed to two key properties of sinusoidal MLPs: smoothness and compactness. These functions are smooth because they arise from the composition of affine maps with the sine function. This work provides theoretical results to justify the compactness property of sinusoidal MLPs and provides control mechanisms in the definition and training of these networks. We propose to study a sinusoidal MLP by expanding it as a harmonic sum. First, we observe that its first layer can be seen as a harmonic dictionary, which we call the input sinusoidal neurons. Then, a hidden layer combines this dictionary using an affine map and modulates the outputs using the sine, this results in a special dictionary of sinusoidal neurons. We prove that each of these sinusoidal neurons expands as a harmonic sum producing a large number of new frequencies expressed as integer linear combinations of the input frequencies. Thus, each hidden neuron produces the same frequencies, and the corresponding amplitudes are completely determined by the hidden affine map. We also provide an upper bound and a way of sorting these amplitudes that can control the resulting approximation, allowing us to truncate the corresponding series. Finally, we present applications for training and initialization of sinusoidal MLPs. Additionally, we show that if the input neurons are periodic, then the entire network will be periodic with the same period. We relate these periodic networks with the Fourier series representation.