SD-$\pi$XL: Generating Low-Resolution Quantized Imagery via Score Distillation

TL;DR

SD-πXL is a novel method that uses score distillation sampling and differentiable image generation to create low-resolution, quantized images like pixel art from prompts or images, with applications in design and fabrication.

Contribution

The paper introduces SD-πXL, a differentiable, score-based approach for generating quantized images with flexible palettes, enabling semantic preservation and practical fabrication applications.

Findings

Outperforms current state-of-the-art in quantized image generation.

Enables transformation of images into low-resolution, semantic-preserving versions.

Demonstrates utility in fabrication tasks like mosaic and embroidery design.

Abstract

Low-resolution quantized imagery, such as pixel art, is seeing a revival in modern applications ranging from video game graphics to digital design and fabrication, where creativity is often bound by a limited palette of elemental units. Despite their growing popularity, the automated generation of quantized images from raw inputs remains a significant challenge, often necessitating intensive manual input. We introduce SD-$\pi$XL, an approach for producing quantized images that employs score distillation sampling in conjunction with a differentiable image generator. Our method enables users to input a prompt and optionally an image for spatial conditioning, set any desired output size $H \times W$, and choose a palette of $n$ colors or elements. Each color corresponds to a distinct class for our generator, which operates on an $H \times W \times n$ tensor. We adopt a softmax approach, computing a convex sum of elements, thus rendering the process differentiable and amenable to backpropagation. We show that employing Gumbel-softmax reparameterization allows for crisp pixel art effects. Unique to our method is the ability to transform input images into low-resolution, quantized versions while retaining their key semantic features. Our experiments validate SD-$\pi$XL's performance in creating visually pleasing and faithful representations, consistently outperforming the current state-of-the-art. Furthermore, we showcase SD-$\pi$XL's practical utility in fabrication through its applications in interlocking brick mosaic, beading and embroidery design.