Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering

TL;DR

Diff3DS is a differentiable rendering framework that generates view-consistent 3D sketches by optimizing 3D curves through 2D image domain gradients, enabling novel sketch generation tasks from text or images.

Contribution

This work introduces Diff3DS, a novel end-to-end differentiable framework for 3D sketch generation that bridges 3D curves and 2D raster images for improved view consistency.

Findings

Effective 3D sketch generation from text and images.

Achieved view consistency in 3D sketches.

Demonstrated promising results on multiple tasks.

Abstract

3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes. However, the creation of 3D sketch often requires users to possess professional artistic skills. Existing research efforts primarily focus on enhancing the ability of interactive sketch generation in 3D virtual systems. In this work, we propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions. Specifically, we perform perspective projection to render the 3D rational B\'ezier curves into 2D curves, which are subsequently converted to a 2D raster image via our customized differentiable rasterizer. Our framework bridges the domains of 3D sketch and raster image, achieving end-toend optimization of 3D sketch through gradients computed in the 2D image domain. Our Diff3DS can enable a series of novel 3D sketch generation tasks, including textto-3D sketch and image-to-3D sketch, supported by the popular distillation-based supervision, such as Score Distillation Sampling (SDS). Extensive experiments have yielded promising results and demonstrated the potential of our framework. Project page is at https://yiboz2001.github.io/Diff3DS/.