Neural Processing of Tri-Plane Hybrid Neural Fields

TL;DR

This paper demonstrates that tri-plane neural fields encode rich information that can be effectively processed with standard deep learning methods, achieving superior task performance compared to large MLPs and nearly matching explicit representations.

Contribution

It introduces a novel approach to process tri-plane neural fields directly, establishing a benchmark and showing superior performance over traditional MLP-based methods.

Findings

Tri-plane neural fields encode rich, processable information.

Processing tri-plane fields yields better task performance than large MLPs.

Achieves near-par performance with explicit 3D representations.

Abstract

Driven by the appealing properties of neural fields for storing and communicating 3D data, the problem of directly processing them to address tasks such as classification and part segmentation has emerged and has been investigated in recent works. Early approaches employ neural fields parameterized by shared networks trained on the whole dataset, achieving good task performance but sacrificing reconstruction quality. To improve the latter, later methods focus on individual neural fields parameterized as large Multi-Layer Perceptrons (MLPs), which are, however, challenging to process due to the high dimensionality of the weight space, intrinsic weight space symmetries, and sensitivity to random initialization. Hence, results turn out significantly inferior to those achieved by processing explicit representations, e.g., point clouds or meshes. In the meantime, hybrid representations, in particular based on tri-planes, have emerged as a more effective and efficient alternative to realize neural fields, but their direct processing has not been investigated yet. In this paper, we show that the tri-plane discrete data structure encodes rich information, which can be effectively processed by standard deep-learning machinery. We define an extensive benchmark covering a diverse set of fields such as occupancy, signed/unsigned distance, and, for the first time, radiance fields. While processing a field with the same reconstruction quality, we achieve task performance far superior to frameworks that process large MLPs and, for the first time, almost on par with architectures handling explicit representations.