HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion

TL;DR

HyperDiffusion introduces a novel diffusion-based generative model operating directly on MLP weights to synthesize high-fidelity implicit neural fields for 3D shapes and animations.

Contribution

It is the first method to perform unconditional generative modeling of implicit neural fields by diffusing in the MLP weight space.

Findings

Successfully generates diverse 3D shapes and animations.

Achieves high-fidelity synthesis of implicit neural representations.

Unifies modeling of static and dynamic 3D data.

Abstract

Implicit neural fields, typically encoded by a multilayer perceptron (MLP) that maps from coordinates (e.g., xyz) to signals (e.g., signed distances), have shown remarkable promise as a high-fidelity and compact representation. However, the lack of a regular and explicit grid structure also makes it challenging to apply generative modeling directly on implicit neural fields in order to synthesize new data. To this end, we propose HyperDiffusion, a novel approach for unconditional generative modeling of implicit neural fields. HyperDiffusion operates directly on MLP weights and generates new neural implicit fields encoded by synthesized MLP parameters. Specifically, a collection of MLPs is first optimized to faithfully represent individual data samples. Subsequently, a diffusion process is trained in this MLP weight space to model the underlying distribution of neural implicit fields. HyperDiffusion enables diffusion modeling over a implicit, compact, and yet high-fidelity representation of complex signals across 3D shapes and 4D mesh animations within one single unified framework.