MangaDiT: Reference-Guided Line Art Colorization with Hierarchical Attention in Diffusion Transformers

TL;DR

MangaDiT is a diffusion transformer-based model that improves reference-guided line art colorization by implicitly discovering semantic correspondences through hierarchical attention, leading to better region-level color consistency.

Contribution

The paper introduces MangaDiT, which employs a hierarchical attention mechanism with dynamic weighting to enhance color alignment without external annotations.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Achieves superior qualitative and quantitative results

Effectively handles pose and motion differences in reference images

Abstract

Recent advances in diffusion models have significantly improved the performance of reference-guided line art colorization. However, existing methods still struggle with region-level color consistency, especially when the reference and target images differ in character pose or motion. Instead of relying on external matching annotations between the reference and target, we propose to discover semantic correspondences implicitly through internal attention mechanisms. In this paper, we present MangaDiT, a powerful model for reference-guided line art colorization based on Diffusion Transformers (DiT). Our model takes both line art and reference images as conditional inputs and introduces a hierarchical attention mechanism with a dynamic attention weighting strategy. This mechanism augments the vanilla attention with an additional context-aware path that leverages pooled spatial features, effectively expanding the model's receptive field and enhancing region-level color alignment. Experiments on two benchmark datasets demonstrate that our method significantly outperforms state-of-the-art approaches, achieving superior performance in both qualitative and quantitative evaluations.