AnimeColor: Reference-based Animation Colorization with Diffusion Transformers

TL;DR

AnimeColor introduces a diffusion transformer-based framework for reference-based animation colorization, achieving superior color accuracy, temporal consistency, and visual quality through innovative components and training strategies.

Contribution

It presents a novel diffusion transformer framework with high-level and low-level color extractors for improved animation colorization.

Findings

Outperforms existing methods in color accuracy

Enhances temporal consistency in animation colorization

Achieves high visual quality in generated animations

Abstract

Animation colorization plays a vital role in animation production, yet existing methods struggle to achieve color accuracy and temporal consistency. To address these challenges, we propose \textbf{AnimeColor}, a novel reference-based animation colorization framework leveraging Diffusion Transformers (DiT). Our approach integrates sketch sequences into a DiT-based video diffusion model, enabling sketch-controlled animation generation. We introduce two key components: a High-level Color Extractor (HCE) to capture semantic color information and a Low-level Color Guider (LCG) to extract fine-grained color details from reference images. These components work synergistically to guide the video diffusion process. Additionally, we employ a multi-stage training strategy to maximize the utilization of reference image color information. Extensive experiments demonstrate that AnimeColor outperforms existing methods in color accuracy, sketch alignment, temporal consistency, and visual quality. Our framework not only advances the state of the art in animation colorization but also provides a practical solution for industrial applications. The code will be made publicly available at \href{https://github.com/IamCreateAI/AnimeColor}{https://github.com/IamCreateAI/AnimeColor}.