Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation

TL;DR

This paper introduces LOTS, a novel framework that combines global sketches with multiple localized text-sketch pairs to improve fashion image generation, validated on a new dataset called Sketchy.

Contribution

The paper proposes LOTS, a multi-level conditioning framework that effectively integrates local and global guidance for fashion image synthesis, along with a new dataset, Sketchy.

Findings

Improves adherence to global structure in generated images

Leverages multiple localized semantic cues for detailed synthesis

Outperforms state-of-the-art methods in fashion image generation

Abstract

Sketches offer designers a concise yet expressive medium for early-stage fashion ideation by specifying structure, silhouette, and spatial relationships, while textual descriptions complement sketches to convey material, color, and stylistic details. Effectively combining textual and visual modalities requires adherence to the sketch visual structure when leveraging the guidance of localized attributes from text. We present LOcalized Text and Sketch with multi-level guidance (LOTS), a framework that enhances fashion image generation by combining global sketch guidance with multiple localized sketch-text pairs. LOTS employs a Multi-level Conditioning Stage to independently encode local features within a shared latent space while maintaining global structural coordination. Then, the Diffusion Pair Guidance stage integrates both local and global conditioning via attention-based guidance within the diffusion model's multi-step denoising process. To validate our method, we develop Sketchy, the first fashion dataset where multiple text-sketch pairs are provided per image. Sketchy provides high-quality, clean sketches with a professional look and consistent structure. To assess robustness beyond this setting, we also include an "in the wild" split with non-expert sketches, featuring higher variability and imperfections. Experiments demonstrate that our method strengthens global structural adherence while leveraging richer localized semantic guidance, achieving improvement over state-of-the-art. The dataset, platform, and code are publicly available.