InstanceAnimator: Multi-Instance Sketch Video Colorization

TL;DR

InstanceAnimator introduces a diffusion transformer framework that significantly improves multi-instance sketch video colorization by enhancing user control, instance alignment, and detail fidelity through novel mechanisms.

Contribution

The paper presents three key innovations—Canvas Guidance Condition, Instance Matching Mechanism, and Adaptive Decoupled Control Module—that collectively advance multi-instance sketch video colorization.

Findings

Achieves superior multi-instance colorization quality.

Enhances user control and flexibility.

Ensures high instance consistency and detail fidelity.

Abstract

We propose InstanceAnimator, a novel Diffusion Transformer framework for multi-instance sketch video colorization. Existing methods suffer from three core limitations: inflexible user control due to heavy reliance on single reference frames, poor instance controllability leading to misalignment in multi-character scenarios, and degraded detail fidelity in fine-grained regions. To address these challenges, we introduce three corresponding innovations. First, a Canvas Guidance Condition eliminates workflow fragmentation by allowing free placement of reference elements and background, enabling unprecedented user flexibility. Second, an Instance Matching Mechanism resolves misalignment by integrating instance features with the sketches, ensuring precise control over multiple characters. Third, an Adaptive Decoupled Control Module enhances detail fidelity by injecting semantic features from characters, backgrounds, and text conditions into the diffusion process. Extensive experiments demonstrate that InstanceAnimator achieves superior multi-instance colorization with enhanced user control, high visual quality, and strong instance consistency.