Weight Space Representation Learning on Diverse NeRF Architectures

TL;DR

This paper introduces a novel framework that learns a unified, architecture-agnostic representation of diverse NeRF models using a graph meta-network and contrastive learning, enabling inference on unseen architectures.

Contribution

It is the first to process and perform inference on diverse NeRF architectures with a single framework, including hash table-based models, using unsupervised contrastive learning.

Findings

Robust performance across 13 diverse NeRF architectures

Effective on classification, retrieval, and language tasks

Outperforms architecture-specific frameworks in generalization

Abstract

Neural Radiance Fields (NeRFs) have emerged as a groundbreaking paradigm for representing 3D objects and scenes by encoding shape and appearance information into the weights of a neural network. Recent studies have demonstrated that these weights can be used as input for frameworks designed to address deep learning tasks; however, such frameworks require NeRFs to adhere to a specific, predefined architecture. In this paper, we introduce the first framework capable of processing NeRFs with diverse architectures and performing inference on architectures unseen at training time. We achieve this by training a Graph Meta-Network within an unsupervised representation learning framework, and show that a contrastive objective is conducive to obtaining an architecture-agnostic latent space. In experiments conducted across 13 NeRF architectures belonging to three families (MLPs, tri-planes, and, for the first time, hash tables), our approach demonstrates robust performance in classification, retrieval, and language tasks involving multiple architectures, even unseen at training time, while also matching or exceeding the results of existing frameworks limited to single architectures. Our code and data are available at https://cvlab-unibo.github.io/gmnerf.