One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control

TL;DR

One4D introduces a unified framework leveraging decoupled LoRA control for high-quality 4D generation and reconstruction, seamlessly handling varying sparsity levels in conditioning data with a novel modality-specific adaptation approach.

Contribution

The paper proposes Decoupled LoRA Control (DLC) with modality-specific adapters and lightweight links, enabling joint RGB and pointmap generation and reconstruction without degrading the base video model.

Findings

High-quality 4D content generation and reconstruction achieved

Effective handling of sparse conditioning frames across tasks

Trained on synthetic and real datasets with modest resources

Abstract

We present One4D, a unified framework for 4D generation and reconstruction that produces dynamic 4D content as synchronized RGB frames and pointmaps. By consistently handling varying sparsities of conditioning frames through a Unified Masked Conditioning (UMC) mechanism, One4D can seamlessly transition between 4D generation from a single image, 4D reconstruction from a full video, and mixed generation and reconstruction from sparse frames. Our framework adapts a powerful video generation model for joint RGB and pointmap generation, with carefully designed network architectures. The commonly used diffusion finetuning strategies for depthmap or pointmap reconstruction often fail on joint RGB and pointmap generation, quickly degrading the base video model. To address this challenge, we introduce Decoupled LoRA Control (DLC), which employs two modality-specific LoRA adapters to form decoupled computation branches for RGB frames and pointmaps, connected by lightweight, zero-initialized control links that gradually learn mutual pixel-level consistency. Trained on a mixture of synthetic and real 4D datasets under modest computational budgets, One4D produces high-quality RGB frames and accurate pointmaps across both generation and reconstruction tasks. This work represents a step toward general, high-quality geometry-based 4D world modeling using video diffusion models. Project page: https://mizhenxing.github.io/One4D