Canonical Fields: Self-Supervised Learning of Pose-Canonicalized Neural Fields

TL;DR

This paper introduces CaFi-Net, a self-supervised approach that canonicalizes 3D poses of neural radiance fields, enabling consistent object representations across categories without relying on canonical datasets.

Contribution

It presents a novel Siamese network architecture for category-level pose canonicalization of neural fields in a self-supervised manner, handling noisy and continuous data.

Findings

Matches or exceeds point cloud-based methods in experiments

Works on a new dataset of 1300 NeRF models across 13 categories

Effectively estimates canonical pose from arbitrary 3D poses

Abstract

Coordinate-based implicit neural networks, or neural fields, have emerged as useful representations of shape and appearance in 3D computer vision. Despite advances, however, it remains challenging to build neural fields for categories of objects without datasets like ShapeNet that provide "canonicalized" object instances that are consistently aligned for their 3D position and orientation (pose). We present Canonical Field Network (CaFi-Net), a self-supervised method to canonicalize the 3D pose of instances from an object category represented as neural fields, specifically neural radiance fields (NeRFs). CaFi-Net directly learns from continuous and noisy radiance fields using a Siamese network architecture that is designed to extract equivariant field features for category-level canonicalization. During inference, our method takes pre-trained neural radiance fields of novel object instances at arbitrary 3D pose and estimates a canonical field with consistent 3D pose across the entire category. Extensive experiments on a new dataset of 1300 NeRF models across 13 object categories show that our method matches or exceeds the performance of 3D point cloud-based methods.