NOVA3D: Normal Aligned Video Diffusion Model for Single Image to 3D Generation

TL;DR

NOVA3D introduces a novel framework that leverages pretrained video diffusion models and geometric alignment techniques to generate high-quality 3D content from a single image with improved multi-view consistency.

Contribution

The paper proposes NOVA3D, integrating 3D priors from video diffusion models and novel attention and fusion algorithms for enhanced 3D generation from a single image.

Findings

Outperforms existing methods in multi-view consistency

Achieves higher texture fidelity and pose accuracy

Demonstrates superior generalization in 3D reconstruction

Abstract

3D AI-generated content (AIGC) has made it increasingly accessible for anyone to become a 3D content creator. While recent methods leverage Score Distillation Sampling to distill 3D objects from pretrained image diffusion models, they often suffer from inadequate 3D priors, leading to insufficient multi-view consistency. In this work, we introduce NOVA3D, an innovative single-image-to-3D generation framework. Our key insight lies in leveraging strong 3D priors from a pretrained video diffusion model and integrating geometric information during multi-view video fine-tuning. To facilitate information exchange between color and geometric domains, we propose the Geometry-Temporal Alignment (GTA) attention mechanism, thereby improving generalization and multi-view consistency. Moreover, we introduce the de-conflict geometry fusion algorithm, which improves texture fidelity by addressing multi-view inaccuracies and resolving discrepancies in pose alignment. Extensive experiments validate the superiority of NOVA3D over existing baselines.