Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs

TL;DR

This paper explores a unifying framework for activation functions in Coordinate-MLPs, demonstrating that non-periodic activations can outperform sinusoidal functions in signal encoding tasks.

Contribution

It broadens the understanding of suitable activation functions for Coordinate-MLPs, proposing non-periodic functions that are more robust than sinusoidal activations.

Findings

Non-periodic activations outperform sinusoidal functions in robustness.

A broader class of activations can effectively encode signals.

Coordinate-MLPs with these activations show high performance and simplicity.

Abstract

Coordinate-MLPs are emerging as an effective tool for modeling multidimensional continuous signals, overcoming many drawbacks associated with discrete grid-based approximations. However, coordinate-MLPs with ReLU activations, in their rudimentary form, demonstrate poor performance in representing signals with high fidelity, promoting the need for positional embedding layers. Recently, Sitzmann et al. proposed a sinusoidal activation function that has the capacity to omit positional embedding from coordinate-MLPs while still preserving high signal fidelity. Despite its potential, ReLUs are still dominating the space of coordinate-MLPs; we speculate that this is due to the hyper-sensitivity of networks -- that employ such sinusoidal activations -- to the initialization schemes. In this paper, we attempt to broaden the current understanding of the effect of activations in coordinate-MLPs, and show that there exists a broader class of activations that are suitable for encoding signals. We affirm that sinusoidal activations are only a single example in this class, and propose several non-periodic functions that empirically demonstrate more robust performance against random initializations than sinusoids. Finally, we advocate for a shift towards coordinate-MLPs that employ these non-traditional activation functions due to their high performance and simplicity.