Random Weight Factorization Improves the Training of Continuous Neural Representations

TL;DR

This paper introduces random weight factorization for coordinate-based MLPs, significantly enhancing training speed and quality in neural representations of signals by improving the loss landscape and enabling adaptive learning rates.

Contribution

It proposes a simple, effective modification to linear layers that accelerates training and improves results across various neural representation tasks.

Findings

Accelerates training of neural representations.

Improves convergence to better local minima.

Enhances ability to recover from poor initializations.

Abstract

Continuous neural representations have recently emerged as a powerful and flexible alternative to classical discretized representations of signals. However, training them to capture fine details in multi-scale signals is difficult and computationally expensive. Here we propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers in coordinate-based multi-layer perceptrons (MLPs) that significantly accelerates and improves their training. We show how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate. This not only helps with mitigating spectral bias, but also allows networks to quickly recover from poor initializations and reach better local minima. We demonstrate how random weight factorization can be leveraged to improve the training of neural representations on a variety of tasks, including image regression, shape representation, computed tomography, inverse rendering, solving partial differential equations, and learning operators between function spaces.