DNF: Unconditional 4D Generation with Dictionary-based Neural Fields

TL;DR

This paper introduces DNF, a novel 4D generative model that uses dictionary learning and neural fields to produce high-fidelity, deformable 4D shapes with disentangled shape and motion, advancing 4D animation generation.

Contribution

The paper presents a new dictionary-based neural field representation for 4D shapes that disentangles shape and motion, enabling efficient, high-quality unconditional 4D generation.

Findings

Effective disentanglement of shape and motion.

High-fidelity 4D shape generation with balanced fidelity and compression.

Successful application of transformer-based diffusion for 4D animation.

Abstract

While remarkable success has been achieved through diffusion-based 3D generative models for shapes, 4D generative modeling remains challenging due to the complexity of object deformations over time. We propose DNF, a new 4D representation for unconditional generative modeling that efficiently models deformable shapes with disentangled shape and motion while capturing high-fidelity details in the deforming objects. To achieve this, we propose a dictionary learning approach to disentangle 4D motion from shape as neural fields. Both shape and motion are represented as learned latent spaces, where each deformable shape is represented by its shape and motion global latent codes, shape-specific coefficient vectors, and shared dictionary information. This captures both shape-specific detail and global shared information in the learned dictionary. Our dictionary-based representation well balances fidelity, contiguity and compression -- combined with a transformer-based diffusion model, our method is able to generate effective, high-fidelity 4D animations.