Skel3D: Skeleton Guided Novel View Synthesis

TL;DR

Skel3D introduces a skeleton-guided diffusion approach for monocular novel view synthesis that improves pose accuracy and multi-view consistency by leveraging object skeletons, outperforming existing methods without explicit 3D models.

Contribution

The paper presents a novel skeleton-guided diffusion method that enhances view synthesis quality by incorporating structural skeleton information into the generative process.

Findings

Significantly improves pose accuracy and multi-view consistency.

Outperforms state-of-the-art NVS methods quantitatively and qualitatively.

Effectively leverages object skeletons without explicit 3D representations.

Abstract

In this paper, we present an approach for monocular open-set novel view synthesis (NVS) that leverages object skeletons to guide the underlying diffusion model. Building upon a baseline that utilizes a pre-trained 2D image generator, our method takes advantage of the Objaverse dataset, which includes animated objects with bone structures. By introducing a skeleton guide layer following the existing ray conditioning normalization (RCN) layer, our approach enhances pose accuracy and multi-view consistency. The skeleton guide layer provides detailed structural information for the generative model, improving the quality of synthesized views. Experimental results demonstrate that our skeleton-guided method significantly enhances consistency and accuracy across diverse object categories within the Objaverse dataset. Our method outperforms existing state-of-the-art NVS techniques both quantitatively and qualitatively, without relying on explicit 3D representations.