DiffBrush:Just Painting the Art by Your Hands
Jiaming Chu, Lei Jin, Tao Wang, Junliang Xing, Jian Zhao

TL;DR
DiffBrush enables precise image editing and generation guided by user sketches within diffusion models, eliminating the need for additional training and allowing control over color, semantics, and object instances.
Contribution
The paper introduces DiffBrush, a novel method that guides diffusion models using user sketches for image editing without extra training, enhancing control and compatibility.
Findings
Achieves accurate image control guided by sketches.
Refines image layout through latent regeneration.
Supports instance-specific image generation from rough masks.
Abstract
The rapid development of image generation and editing algorithms in recent years has enabled ordinary user to produce realistic images. However, the current AI painting ecosystem predominantly relies on text-driven diffusion models (T2I), which pose challenges in accurately capturing user requirements. Furthermore, achieving compatibility with other modalities incurs substantial training costs. To this end, we introduce DiffBrush, which is compatible with T2I models and allows users to draw and edit images. By manipulating and adapting the internal representation of the diffusion model, DiffBrush guides the model-generated images to converge towards the user's hand-drawn sketches for user's specific needs without additional training. DiffBrush achieves control over the color, semantic, and instance of objects in images by continuously guiding the latent and instance-level attention map…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsSoftmax · Attention Is All You Need · Diffusion
