StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback

TL;DR

StableSketcher is a novel framework that enhances diffusion models to generate high-fidelity, prompt-aligned pixel-based sketches by fine-tuning autoencoders and incorporating visual question answering feedback.

Contribution

It introduces a new reward function based on visual question answering and a dataset with sketches, captions, and Q&A pairs, advancing sketch generation quality and evaluation.

Findings

StableSketcher produces sketches with better stylistic fidelity.

It achieves improved prompt alignment over the Stable Diffusion baseline.

The new dataset SketchDUO addresses limitations of existing sketch datasets.

Abstract

Although recent advancements in diffusion models have significantly enriched the quality of generated images, challenges remain in synthesizing pixel-based human-drawn sketches, a representative example of abstract expression. To combat these challenges, we propose StableSketcher, a novel framework that empowers diffusion models to generate hand-drawn sketches with high prompt fidelity. Within this framework, we fine-tune the variational autoencoder to optimize latent decoding, enabling it to better capture the characteristics of sketches. In parallel, we integrate a new reward function for reinforcement learning based on visual question answering, which improves text-image alignment and semantic consistency. Extensive experiments demonstrate that StableSketcher generates sketches with improved stylistic fidelity, achieving better alignment with prompts compared to the Stable Diffusion baseline. Additionally, we introduce SketchDUO, to the best of our knowledge, the first dataset comprising instance-level sketches paired with captions and question-answer pairs, thereby addressing the limitations of existing datasets that rely on image-label pairs. Our code and dataset will be made publicly available upon acceptance. Project page: https://zihos.github.io/StableSketcher