Scones: Towards Conversational Authoring of Sketches

TL;DR

Scones is a novel AI system that enables users to create and refine sketches from text instructions through iterative, machine-learning-driven scene generation, enhancing design communication.

Contribution

Introduces Scones, a deep-learning system for iterative sketch authoring from natural language, with a mask-conditioned model for pose specification, surpassing existing methods.

Findings

Scones outperforms state-of-the-art in scene modification tasks.

Participants enjoyed the iterative drawing experience with Scones.

User feedback suggests potential for further feature development.

Abstract

Iteratively refining and critiquing sketches are crucial steps to developing effective designs. We introduce Scones, a mixed-initiative, machine-learning-driven system that enables users to iteratively author sketches from text instructions. Scones is a novel deep-learning-based system that iteratively generates scenes of sketched objects composed with semantic specifications from natural language. Scones exceeds state-of-the-art performance on a text-based scene modification task, and introduces a mask-conditioned sketching model that can generate sketches with poses specified by high-level scene information. In an exploratory user evaluation of Scones, participants reported enjoying an iterative drawing task with Scones, and suggested additional features for further applications. We believe Scones is an early step towards automated, intelligent systems that support human-in-the-loop applications for communicating ideas through sketching in art and design.