X-Ray: A Sequential 3D Representation For Generation

TL;DR

X-Ray introduces a novel 3D sequential surface-based representation inspired by x-ray scans, enabling efficient and accurate 3D object generation from images using a multi-frame video format and diffusion models.

Contribution

The paper presents a new 3D representation method called X-Ray that captures surface details in a sequential format, improving 3D generation accuracy from single images.

Findings

Achieves state-of-the-art accuracy in 3D generation.

Efficiently captures complete surface details from a single image.

Demonstrates practical applicability in 3D model synthesis.

Abstract

We introduce X-Ray, a novel 3D sequential representation inspired by the penetrability of x-ray scans. X-Ray transforms a 3D object into a series of surface frames at different layers, making it suitable for generating 3D models from images. Our method utilizes ray casting from the camera center to capture geometric and textured details, including depth, normal, and color, across all intersected surfaces. This process efficiently condenses the whole 3D object into a multi-frame video format, motivating the utilize of a network architecture similar to those in video diffusion models. This design ensures an efficient 3D representation by focusing solely on surface information. Also, we propose a two-stage pipeline to generate 3D objects from X-Ray Diffusion Model and Upsampler. We demonstrate the practicality and adaptability of our X-Ray representation by synthesizing the complete visible and hidden surfaces of a 3D object from a single input image. Experimental results reveal the state-of-the-art superiority of our representation in enhancing the accuracy of 3D generation, paving the way for new 3D representation research and practical applications.