EscherNet++: Simultaneous Amodal Completion and Scalable View Synthesis through Masked Fine-Tuning and Enhanced Feed-Forward 3D Reconstruction

TL;DR

EscherNet++ is a novel end-to-end diffusion model that performs amodal completion and scalable view synthesis in a zero-shot manner, significantly reducing computation time and improving 3D reconstruction quality.

Contribution

The paper introduces masked fine-tuning in a diffusion model for simultaneous view synthesis and amodal completion, enabling end-to-end training and integration with existing models without extra training.

Findings

Achieves 95% reduction in reconstruction time.

Improves PSNR by 3.9 and Volume IoU by 0.28 on occluded tasks.

Generalizes well to real-world occluded reconstructions.

Abstract

We propose EscherNet++, a masked fine-tuned diffusion model that can synthesize novel views of objects in a zero-shot manner with amodal completion ability. Existing approaches utilize multiple stages and complex pipelines to first hallucinate missing parts of the image and then perform novel view synthesis, which fail to consider cross-view dependencies and require redundant storage and computing for separate stages. Instead, we apply masked fine-tuning including input-level and feature-level masking to enable an end-to-end model with the improved ability to synthesize novel views and conduct amodal completion. In addition, we empirically integrate our model with other feed-forward image-to-mesh models without extra training and achieve competitive results with reconstruction time decreased by 95%, thanks to its ability to synthesize arbitrary query views. Our method's scalable nature further enhances fast 3D reconstruction. Despite fine-tuning on a smaller dataset and batch size, our method achieves state-of-the-art results, improving PSNR by 3.9 and Volume IoU by 0.28 on occluded tasks in 10-input settings, while also generalizing to real-world occluded reconstruction.