Image Referenced Sketch Colorization Based on Animation Creation Workflow
Dingkun Yan, Xinrui Wang, Zhuoru Li, Suguru Saito, Yusuke Iwasawa,, Yutaka Matsuo, Jiaxian Guo

TL;DR
This paper introduces a diffusion-based sketch colorization framework inspired by animation workflows, effectively using spatial guidance and RGB references to produce artifact-free, high-quality colorized images with independent foreground and background control.
Contribution
The proposed method uniquely integrates spatial masks and a split cross-attention mechanism with LoRA modules, enabling independent foreground and background colorization and artifact elimination.
Findings
Outperforms existing methods in artifact-free, high-quality colorization
Effectively handles geometric mismatched references
Demonstrates robustness through extensive experiments and user studies
Abstract
Sketch colorization plays an important role in animation and digital illustration production tasks. However, existing methods still meet problems in that text-guided methods fail to provide accurate color and style reference, hint-guided methods still involve manual operation, and image-referenced methods are prone to cause artifacts. To address these limitations, we propose a diffusion-based framework inspired by real-world animation production workflows. Our approach leverages the sketch as the spatial guidance and an RGB image as the color reference, and separately extracts foreground and background from the reference image with spatial masks. Particularly, we introduce a split cross-attention mechanism with LoRA (Low-Rank Adaptation) modules. They are trained separately with foreground and background regions to control the corresponding embeddings for keys and values in…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Computer Graphics and Visualization Techniques · 3D Shape Modeling and Analysis
MethodsDiffusion · Colorization
