Sketch-based Creativity Support Tools using Deep Learning

TL;DR

This paper explores deep learning methods for sketch-based creative tools, including dataset creation, sketch retrieval, and conversational sketching, advancing human-computer interaction in creative applications.

Contribution

It introduces a new paired dataset, a sketch-based retrieval system, and a conversational sketching interface, advancing deep learning applications in creative sketching tools.

Findings

Successful development of a new sketch-UI dataset

Effective sketch retrieval system demonstrated

Qualitative and quantitative results show system potential

Abstract

Sketching is a natural and effective visual communication medium commonly used in creative processes. Recent developments in deep-learning models drastically improved machines' ability in understanding and generating visual content. An exciting area of development explores deep-learning approaches used to model human sketches, opening opportunities for creative applications. This chapter describes three fundamental steps in developing deep-learning-driven creativity support tools that consumes and generates sketches: 1) a data collection effort that generated a new paired dataset between sketches and mobile user interfaces; 2) a sketch-based user interface retrieval system adapted from state-of-the-art computer vision techniques; and, 3) a conversational sketching system that supports the novel interaction of a natural-language-based sketch/critique authoring process. In this chapter, we survey relevant prior work in both the deep-learning and human-computer-interaction communities, document the data collection process and the systems' architectures in detail, present qualitative and quantitative results, and paint the landscape of several future research directions in this exciting area.