Enhancing Reference-based Sketch Colorization via Separating Reference Representations

TL;DR

This paper proposes a modular framework for reference-based sketch colorization that separates reference representations to improve quality and robustness against misaligned references, outperforming existing methods.

Contribution

It introduces a novel approach that decomposes colorization into stages with distinct reference representations, enhancing visual quality and reference similarity.

Findings

Outperforms existing methods in qualitative and quantitative evaluations

Reduces artifacts caused by reference-sketch misalignment

Enables flexible inference modes for various use cases

Abstract

Reference-based sketch colorization methods have garnered significant attention for the potential application in animation and digital illustration production. However, most existing methods are trained with image triplets of sketch, reference, and ground truth that are semantically and spatially similar, while real-world references and sketches often exhibit substantial misalignment. This mismatch in data distribution between training and inference leads to overfitting, consequently resulting in artifacts and signif- icant quality degradation in colorization results. To address this issue, we conduct an in-depth analysis of the reference representations, defined as the intermedium to transfer information from reference to sketch. Building on this analysis, we introduce a novel framework that leverages distinct reference representations to optimize different aspects of the colorization process. Our approach decomposes colorization into modular stages, al- lowing region-specific reference injection to enhance visual quality and reference similarity while mitigating spatial artifacts. Specifically, we first train a backbone network guided by high-level semantic embeddings. We then introduce a background encoder and a style encoder, trained in separate stages, to enhance low-level feature transfer and improve reference similar- ity. This design also enables flexible inference modes suited for a variety of use cases. Extensive qualitative and quantitative evaluations, together with a user study, demonstrate the superior performance of our proposed method compared to existing approaches. Code and pre-trained weight will be made publicly available upon paper acceptance.