VQ-SGen: A Vector Quantized Stroke Representation for Creative Sketch Generation

TL;DR

VQ-SGen introduces a vector-quantized stroke representation for sketch generation, capturing intrinsic stroke relationships and enabling high-quality, flexible, and conditioned sketch creation.

Contribution

The paper proposes a novel two-stage framework using VQ stroke representation and an auto-decoding Transformer for improved sketch generation.

Findings

Outperforms state-of-the-art on CreativeSketch dataset

Enables text and class-conditioned sketch generation

Facilitates sketch completion with high fidelity

Abstract

This paper presents VQ-SGen, a novel algorithm for high-quality creative sketch generation. Recent approaches have framed the task as pixel-based generation either as a whole or part-by-part, neglecting the intrinsic and contextual relationships among individual strokes, such as the shape and spatial positioning of both proximal and distant strokes. To overcome these limitations, we propose treating each stroke within a sketch as an entity and introducing a vector-quantized (VQ) stroke representation for fine-grained sketch generation. Our method follows a two-stage framework - in stage one, we decouple each stroke's shape and location information to ensure the VQ representation prioritizes stroke shape learning. In stage two, we feed the precise and compact representation into an auto-decoding Transformer to incorporate stroke semantics, positions, and shapes into the generation process. By utilizing tokenized stroke representation, our approach generates strokes with high fidelity and facilitates novel applications, such as text or class label conditioned generation and sketch completion. Comprehensive experiments demonstrate our method surpasses existing state-of-the-art techniques on the CreativeSketch dataset, underscoring its effectiveness.