Correspondence Distillation from NeRF-based GAN

TL;DR

This paper introduces a novel method for establishing dense correspondences across NeRF-based GANs by leveraging semantic and structural priors, enabling improved 3D alignment and downstream applications.

Contribution

It proposes a new approach that uses priors from a pre-trained NeRF-GAN to achieve accurate, smooth, and robust dense correspondences across NeRFs without ground-truth annotations.

Findings

Achieves accurate 3D dense correspondence across NeRFs.

Enables effective texture transfer and downstream tasks.

Demonstrates robustness and smoothness in correspondence mapping.

Abstract

The neural radiance field (NeRF) has shown promising results in preserving the fine details of objects and scenes. However, unlike mesh-based representations, it remains an open problem to build dense correspondences across different NeRFs of the same category, which is essential in many downstream tasks. The main difficulties of this problem lie in the implicit nature of NeRF and the lack of ground-truth correspondence annotations. In this paper, we show it is possible to bypass these challenges by leveraging the rich semantics and structural priors encapsulated in a pre-trained NeRF-based GAN. Specifically, we exploit such priors from three aspects, namely 1) a dual deformation field that takes latent codes as global structural indicators, 2) a learning objective that regards generator features as geometric-aware local descriptors, and 3) a source of infinite object-specific NeRF samples. Our experiments demonstrate that such priors lead to 3D dense correspondence that is accurate, smooth, and robust. We also show that established dense correspondence across NeRFs can effectively enable many NeRF-based downstream applications such as texture transfer.