Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day

TL;DR

Efficient-3DiM introduces a framework that significantly reduces training time for single-image novel view synthesis from 10 days to under 1 day, maintaining high quality and generalizability.

Contribution

The paper presents pragmatic strategies to accelerate diffusion-based 3D view synthesis training, making it feasible within a day on standard GPU hardware.

Findings

Training time reduced from 10 days to less than 1 day.

Achieves high-quality novel view synthesis with limited training overhead.

Demonstrates strong generalizability across different scenes.

Abstract

The task of novel view synthesis aims to generate unseen perspectives of an object or scene from a limited set of input images. Nevertheless, synthesizing novel views from a single image still remains a significant challenge in the realm of computer vision. Previous approaches tackle this problem by adopting mesh prediction, multi-plain image construction, or more advanced techniques such as neural radiance fields. Recently, a pre-trained diffusion model that is specifically designed for 2D image synthesis has demonstrated its capability in producing photorealistic novel views, if sufficiently optimized on a 3D finetuning task. Although the fidelity and generalizability are greatly improved, training such a powerful diffusion model requires a vast volume of training data and model parameters, resulting in a notoriously long time and high computational costs. To tackle this issue, we propose Efficient-3DiM, a simple but effective framework to learn a single-image novel-view synthesizer. Motivated by our in-depth analysis of the inference process of diffusion models, we propose several pragmatic strategies to reduce the training overhead to a manageable scale, including a crafted timestep sampling strategy, a superior 3D feature extractor, and an enhanced training scheme. When combined, our framework is able to reduce the total training time from 10 days to less than 1 day, significantly accelerating the training process under the same computational platform (one instance with 8 Nvidia A100 GPUs). Comprehensive experiments are conducted to demonstrate the efficiency and generalizability of our proposed method.