Deep Learning on Object-centric 3D Neural Fields

TL;DR

This paper introduces nf2vec, a framework that creates compact latent representations of 3D neural fields, enabling their integration into deep learning pipelines for various 3D tasks without converting to traditional formats.

Contribution

The paper presents nf2vec, a novel method for embedding neural fields directly, facilitating end-to-end deep learning on 3D data represented by neural fields.

Findings

nf2vec effectively embeds 3D neural fields for downstream tasks

The approach works with various neural fields including distance, occupancy, and radiance fields

Embeddings enable deep learning models to process 3D data directly from neural fields

Abstract

In recent years, Neural Fields (NFs) have emerged as an effective tool for encoding diverse continuous signals such as images, videos, audio, and 3D shapes. When applied to 3D data, NFs offer a solution to the fragmentation and limitations associated with prevalent discrete representations. However, given that NFs are essentially neural networks, it remains unclear whether and how they can be seamlessly integrated into deep learning pipelines for solving downstream tasks. This paper addresses this research problem and introduces nf2vec, a framework capable of generating a compact latent representation for an input NF in a single inference pass. We demonstrate that nf2vec effectively embeds 3D objects represented by the input NFs and showcase how the resulting embeddings can be employed in deep learning pipelines to successfully address various tasks, all while processing exclusively NFs. We test this framework on several NFs used to represent 3D surfaces, such as unsigned/signed distance and occupancy fields. Moreover, we demonstrate the effectiveness of our approach with more complex NFs that encompass both geometry and appearance of 3D objects such as neural radiance fields.