LatentColorization: Latent Diffusion-Based Speaker Video Colorization

TL;DR

This paper introduces LatentColorization, a novel latent diffusion-based method for video colorization that ensures temporal consistency and outperforms existing techniques in quality and user preference.

Contribution

The paper presents a fine-tuned latent diffusion model specifically designed for video colorization with temporal coherence, a novel approach in this domain.

Findings

Improved temporal consistency in video colorization.

Higher image quality metrics compared to existing methods.

Users preferred the proposed approach in subjective evaluations.

Abstract

While current research predominantly focuses on image-based colorization, the domain of video-based colorization remains relatively unexplored. Most existing video colorization techniques operate on a frame-by-frame basis, often overlooking the critical aspect of temporal coherence between successive frames. This approach can result in inconsistencies across frames, leading to undesirable effects like flickering or abrupt color transitions between frames. To address these challenges, we harness the generative capabilities of a fine-tuned latent diffusion model designed specifically for video colorization, introducing a novel solution for achieving temporal consistency in video colorization, as well as demonstrating strong improvements on established image quality metrics compared to other existing methods. Furthermore, we perform a subjective study, where users preferred our approach to the existing state of the art. Our dataset encompasses a combination of conventional datasets and videos from television/movies. In short, by leveraging the power of a fine-tuned latent diffusion-based colorization system with a temporal consistency mechanism, we can improve the performance of automatic video colorization by addressing the challenges of temporal inconsistency. A short demonstration of our results can be seen in some example videos available at https://youtu.be/vDbzsZdFuxM.