Painter: Teaching Auto-regressive Language Models to Draw Sketches

TL;DR

Painter demonstrates how large language models can be fine-tuned to generate detailed sketches from text prompts, enabling diverse image creation and editing tasks in an auto-regressive manner.

Contribution

This work introduces Painter, a novel approach that adapts pre-trained LLMs for sketch generation from text, combining language understanding with image synthesis capabilities.

Findings

Painter can generate sketches from textual descriptions.

It can remove objects and detect objects within sketches.

Results show promising potential for LLMs in image generation tasks.

Abstract

Large language models (LLMs) have made tremendous progress in natural language understanding and they have also been successfully adopted in other domains such as computer vision, robotics, reinforcement learning, etc. In this work, we apply LLMs to image generation tasks by directly generating the virtual brush strokes to paint an image. We present Painter, an LLM that can convert user prompts in text description format to sketches by generating the corresponding brush strokes in an auto-regressive way. We construct Painter based on off-the-shelf LLM that is pre-trained on a large text corpus, by fine-tuning it on the new task while preserving language understanding capabilities. We create a dataset of diverse multi-object sketches paired with textual prompts that covers several object types and tasks. Painter can generate sketches from text descriptions, remove objects from canvas, and detect and classify objects in sketches. Although this is an unprecedented pioneering work in using LLMs for auto-regressive image generation, the results are very encouraging.