Hybrid Fourier Score Distillation for Efficient One Image to 3D Object Generation

TL;DR

This paper introduces hy-FSD, a hybrid Fourier Score Distillation method that combines 3D geometric priors and 2D appearance priors in different domains to enable fast, high-quality 3D object generation from a single image.

Contribution

The paper proposes a novel hybrid Fourier Score Distillation approach that effectively integrates 3D and 2D priors for efficient single-image 3D object generation.

Findings

Enables high-quality 3D object creation within one minute.

Achieves rapid convergence and visually appealing results.

Outperforms existing methods in efficiency and quality.

Abstract

Single image-to-3D generation is pivotal for crafting controllable 3D assets. Given its under-constrained nature, we attempt to leverage 3D geometric priors from a novel view diffusion model and 2D appearance priors from an image generation model to guide the optimization process. We note that there is a disparity between the generation priors of these two diffusion models, leading to their different appearance outputs. Specifically, image generation models tend to deliver more detailed visuals, whereas novel view models produce consistent yet over-smooth results across different views. Directly combining them leads to suboptimal effects due to their appearance conflicts. Hence, we propose a 2D-3D hybrid Fourier Score Distillation objective function, hy-FSD. It optimizes 3D Gaussians using 3D priors in spatial domain to ensure geometric consistency, while exploiting 2D priors in the frequency domain through Fourier transform for better visual quality. hy-FSD can be integrated into existing 3D generation methods and produce significant performance gains. With this technique, we further develop an image-to-3D generation pipeline to create high-quality 3D objects within one minute, named Fourier123. Extensive experiments demonstrate that Fourier123 excels in efficient generation with rapid convergence speed and visually-friendly generation results.