SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition

TL;DR

SketchGPT introduces a sequence-to-sequence autoregressive framework that simplifies sketch data into primitive sequences, enabling effective sketch generation, completion, and recognition with competitive results and human evaluation.

Contribution

The paper proposes a novel sketch representation and modeling approach that enhances autoregressive sketch generation and recognition, overcoming previous challenges with continuous stroke data.

Findings

SketchGPT can generate diverse sketches with high quality.

The model achieves competitive performance compared to state-of-the-art methods.

Human evaluations confirm the effectiveness of SketchGPT in sketch recognition.

Abstract

We present SketchGPT, a flexible framework that employs a sequence-to-sequence autoregressive model for sketch generation, and completion, and an interpretation case study for sketch recognition. By mapping complex sketches into simplified sequences of abstract primitives, our approach significantly streamlines the input for autoregressive modeling. SketchGPT leverages the next token prediction objective strategy to understand sketch patterns, facilitating the creation and completion of drawings and also categorizing them accurately. This proposed sketch representation strategy aids in overcoming existing challenges of autoregressive modeling for continuous stroke data, enabling smoother model training and competitive performance. Our findings exhibit SketchGPT's capability to generate a diverse variety of drawings by adding both qualitative and quantitative comparisons with existing state-of-the-art, along with a comprehensive human evaluation study. The code and pretrained models will be released on our official GitHub.