Multi-Head ReLU Implicit Neural Representation Networks

TL;DR

This paper introduces a multi-head ReLU-based neural network architecture for implicit neural representations that mitigates spectral bias, enhances generalization, and reduces computational cost compared to existing methods.

Contribution

It proposes a novel multi-head MLP structure that captures both global and local features, addressing spectral bias and improving efficiency in INR tasks.

Findings

Outperforms existing INR methods in accuracy.

Exhibits better generalization capabilities.

Requires less computational resources.

Abstract

In this paper, a novel multi-head multi-layer perceptron (MLP) structure is presented for implicit neural representation (INR). Since conventional rectified linear unit (ReLU) networks are shown to exhibit spectral bias towards learning low-frequency features of the signal, we aim at mitigating this defect by taking advantage of the local structure of the signals. To be more specific, an MLP is used to capture the global features of the underlying generator function of the desired signal. Then, several heads are utilized to reconstruct disjoint local features of the signal, and to reduce the computational complexity, sparse layers are deployed for attaching heads to the body. Through various experiments, we show that the proposed model does not suffer from the special bias of conventional ReLU networks and has superior generalization capabilities. Finally, simulation results confirm that the proposed multi-head structure outperforms existing INR methods with considerably less computational cost.