Canonical Factors for Hybrid Neural Fields

TL;DR

This paper introduces TILTED, a method that learns canonical transformations to reduce biases in hybrid neural fields, leading to more efficient, robust, and compact representations for various 3D reconstruction tasks.

Contribution

It characterizes biases in factored neural fields and proposes a novel approach to learn canonicalizing transformations, significantly improving efficiency and quality.

Findings

TILTED improves reconstruction quality, robustness, and compactness.

It achieves comparable performance to larger models with fewer parameters.

The method reveals weaknesses in current neural field evaluation procedures.

Abstract

Factored feature volumes offer a simple way to build more compact, efficient, and intepretable neural fields, but also introduce biases that are not necessarily beneficial for real-world data. In this work, we (1) characterize the undesirable biases that these architectures have for axis-aligned signals -- they can lead to radiance field reconstruction differences of as high as 2 PSNR -- and (2) explore how learning a set of canonicalizing transformations can improve representations by removing these biases. We prove in a two-dimensional model problem that simultaneously learning these transformations together with scene appearance succeeds with drastically improved efficiency. We validate the resulting architectures, which we call TILTED, using image, signed distance, and radiance field reconstruction tasks, where we observe improvements across quality, robustness, compactness, and runtime. Results demonstrate that TILTED can enable capabilities comparable to baselines that are 2x larger, while highlighting weaknesses of neural field evaluation procedures.