LSS3D: Learnable Spatial Shifting for Consistent and High-Quality 3D Generation from Single-Image

TL;DR

LSS3D introduces learnable spatial shifting for multi-view 3D generation from a single image, improving consistency, geometric detail, and robustness to various input angles.

Contribution

The paper proposes a novel learnable spatial shifting mechanism to explicitly address multiview inconsistencies and non-frontal inputs in 3D generation from single images.

Findings

Achieves superior geometric and texture quality in 3D models.

Demonstrates robustness to diverse input viewpoints.

Provides a comprehensive evaluation pipeline for performance comparison.

Abstract

Recently, multi-view diffusion-based 3D generation methods have gained significant attention. However, these methods often suffer from shape and texture misalignment across generated multi-view images, leading to low-quality 3D generation results, such as incomplete geometric details and textural ghosting. Some methods are mainly optimized for the frontal perspective and exhibit poor robustness to oblique perspective inputs. In this paper, to tackle the above challenges, we propose a high-quality image-to-3D approach, named LSS3D, with learnable spatial shifting to explicitly and effectively handle the multiview inconsistencies and non-frontal input view. Specifically, we assign learnable spatial shifting parameters to each view, and adjust each view towards a spatially consistent target, guided by the reconstructed mesh, resulting in high-quality 3D generation with more complete geometric details and clean textures. Besides, we include the input view as an extra constraint for the optimization, further enhancing robustness to non-frontal input angles, especially for elevated viewpoint inputs. We also provide a comprehensive quantitative evaluation pipeline that can contribute to the community in performance comparisons. Extensive experiments demonstrate that our method consistently achieves leading results in both geometric and texture evaluation metrics across more flexible input viewpoints.