Sketch-1-to-3: One Single Sketch to 3D Detailed Face Reconstruction

TL;DR

This paper introduces Sketch-1-to-3, a novel deep learning framework that reconstructs detailed 3D faces from single sketches, overcoming modality gaps and limited data challenges, and achieves state-of-the-art results.

Contribution

The paper presents a new framework with a GCTD module and domain adaptation techniques for high-fidelity 3D face reconstruction from sketches, along with new datasets for evaluation.

Findings

Achieves state-of-the-art performance in sketch-based 3D face reconstruction.

Introduces the GCTD module for better feature extraction from sketches.

Provides new datasets: SketchFaces and Syn-SketchFaces.

Abstract

3D face reconstruction from a single sketch is a critical yet underexplored task with significant practical applications. The primary challenges stem from the substantial modality gap between 2D sketches and 3D facial structures, including: (1) accurately extracting facial keypoints from 2D sketches; (2) preserving diverse facial expressions and fine-grained texture details; and (3) training a high-performing model with limited data. In this paper, we propose Sketch-1-to-3, a novel framework for realistic 3D face reconstruction from a single sketch, to address these challenges. Specifically, we first introduce the Geometric Contour and Texture Detail (GCTD) module, which enhances the extraction of geometric contours and texture details from facial sketches. Additionally, we design a deep learning architecture with a domain adaptation module and a tailored loss function to align sketches with the 3D facial space, enabling high-fidelity expression and texture reconstruction. To facilitate evaluation and further research, we construct SketchFaces, a real hand-drawn facial sketch dataset, and Syn-SketchFaces, a synthetic facial sketch dataset. Extensive experiments demonstrate that Sketch-1-to-3 achieves state-of-the-art performance in sketch-based 3D face reconstruction.