SHAD3S: A model to Sketch, Shade and Shadow

TL;DR

SHAD3S is an AI system that automatically generates realistic hatch patterns on 3D shapes from sketches, aiding artists in shading and form exploration without requiring 3D models.

Contribution

It introduces a novel CGAN-based pipeline for automatic hatching on 3D shapes using only 2D sketches and contextual info, without 3D access.

Findings

The model generalizes well across diverse shapes and styles.

Generated hatch patterns are as diverse as real human sketches.

The system provides an interactive tool integrated with GIMP.

Abstract

Hatching is a common method used by artists to accentuate the third dimension of a sketch, and to illuminate the scene. Our system SHAD3S attempts to compete with a human at hatching generic three-dimensional (3D) shapes, and also tries to assist her in a form exploration exercise. The novelty of our approach lies in the fact that we make no assumptions about the input other than that it represents a 3D shape, and yet, given a contextual information of illumination and texture, we synthesise an accurate hatch pattern over the sketch, without access to 3D or pseudo 3D. In the process, we contribute towards a) a cheap yet effective method to synthesise a sufficiently large high fidelity dataset, pertinent to task; b) creating a pipeline with conditional generative adversarial network (CGAN); and c) creating an interactive utility with GIMP, that is a tool for artists to engage with automated hatching or a form-exploration exercise. User evaluation of the tool suggests that the model performance does generalise satisfactorily over diverse input, both in terms of style as well as shape. A simple comparison of inception scores suggest that the generated distribution is as diverse as the ground truth.