Towards High-resolution and Disentangled Reference-based Sketch Colorization

TL;DR

This paper introduces a dual-branch framework with a Gram Regularization Loss and an anime-specific Tagger Network to improve high-resolution, disentangled reference-based sketch colorization by directly addressing distribution shift issues.

Contribution

It presents a novel dual-branch model with regularization and fine-grained control mechanisms, achieving state-of-the-art results in sketch colorization.

Findings

Superior colorization quality and resolution

Enhanced controllability over color transfer

Outperforms existing methods in quantitative and qualitative metrics

Abstract

Sketch colorization is a critical task for automating and assisting in the creation of animations and digital illustrations. Previous research identified the primary difficulty as the distribution shift between semantically aligned training data and highly diverse test data, and focused on mitigating the artifacts caused by the distribution shift instead of fundamentally resolving the problem. In this paper, we present a framework that directly minimizes the distribution shift, thereby achieving superior quality, resolution, and controllability of colorization. We propose a dual-branch framework to explicitly model the data distributions of the training process and inference process with a semantic-aligned branch and a semantic-misaligned branch, respectively. A Gram Regularization Loss is applied across the feature maps of both branches, effectively enforcing cross-domain distribution coherence and stability. Furthermore, we adopt an anime-specific Tagger Network to extract fine-grained attributions from reference images and modulate SDXL's conditional encoders to ensure precise control, and a plugin module to enhance texture transfer. Quantitative and qualitative comparisons, alongside user studies, confirm that our method effectively overcomes the distribution shift challenge, establishing State-of-the-Art performance across both quality and controllability metrics. Ablation study reveals the influence of each component.