The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy

TL;DR

This paper identifies a fundamental rank deficiency in implicit neural representations caused by the inlet rank collapse phenomenon and proposes a unified diagnostic and remedy to enhance their expressive capacity.

Contribution

It introduces a layer-wise NTK decomposition framework to diagnose inlet rank collapse and proposes a rank-expanding initialization to improve INR performance without architectural changes.

Findings

Rank-expanding initialization improves signal reconstruction quality.

Structural diagnosis unifies understanding of existing techniques like PE, SIREN, and BN.

Method enables high-fidelity reconstructions with standard MLPs.

Abstract

Implicit Neural Representations (INRs) have revolutionized continuous signal modeling, yet they struggle to recover fine-grained details within finite training budgets. While empirical techniques, such as positional encoding (PE), sinusoidal activations (SIREN), and batch normalization (BN), effectively mitigate this, their theoretical justifications are predominantly post hoc, focusing on the global NTK spectrum only after modifications are applied. In this work, we reverse this paradigm by introducing a structural diagnostic framework. By performing a layer-wise decomposition of the NTK, we mathematically identify the ``Inlet Rank Collapse'': a phenomenon where the low-dimensional input coordinates fail to span the high-dimensional embedding space, creating a fundamental rank deficiency at the first layer that acts as an expressive bottleneck for the entire network. This framework provides a unified perspective to re-interpret PE, SIREN, and BN as different forms of rank restoration. Guided by this diagnosis, we derive a Rank-Expanding Initialization, a minimalist remedy that ensures the representation rank scales with the layer width without architectural modifications or computational overhead. Our results demonstrate that this principled remedy enables standard MLPs to achieve high-fidelity reconstructions, proving that the key to empowering INRs lies in the structural optimization of the initial rank propagation to effectively populate the latent space.