LUCSS: Language-based User-customized Colourization of Scene Sketches

TL;DR

LUCSS is a deep learning system that enables interactive, language-based colorization of scene sketches by understanding their semantics and allowing user edits through text descriptions.

Contribution

It introduces a novel multi-module neural network framework that combines sketch segmentation, captioning, and interactive colorization based on text, enabling user-guided scene sketch coloring.

Findings

Effective segmentation and captioning of scene sketches.

Successful interactive colorization based on user-modified captions.

Demonstrated superiority over alternative methods.

Abstract

We introduce LUCSS, a language-based system for interactive col- orization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It con- sists of three sequential modules. First, given a scene sketch, the segmenta- tion module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Fi- nally, the interactive colorization module allows users to edit the caption and produce colored images based on the altered caption. Our experiments show the effectiveness of our approach and the desirability of its compo- nents to alternative choices.