ChromaDistill: Colorizing Monochrome Radiance Fields with Knowledge Distillation

TL;DR

ChromaDistill introduces a novel distillation approach to colorize 3D radiance fields from grayscale images, achieving consistent, high-quality colorization without additional computational costs, applicable to various 3D representations and scenarios.

Contribution

The paper presents a new distillation-based method for colorizing 3D scenes that transfers knowledge from image colorization models without increasing inference complexity.

Findings

Produces high-quality, consistent colorized views for diverse scenes

Applicable to NeRF and 3DGS representations without extra inference costs

Effective in applications like IR multi-view and legacy grayscale sequences

Abstract

Colorization is a well-explored problem in the domains of image and video processing. However, extending colorization to 3D scenes presents significant challenges. Recent Neural Radiance Field (NeRF) and Gaussian-Splatting(3DGS) methods enable high-quality novel-view synthesis for multi-view images. However, the question arises: How can we colorize these 3D representations? This work presents a method for synthesizing colorized novel views from input grayscale multi-view images. Using image or video colorization methods to colorize novel views from these 3D representations naively will yield output with severe inconsistencies. We introduce a novel method to use powerful image colorization models for colorizing 3D representations. We propose a distillation-based method that transfers color from these networks trained on natural images to the target 3D representation. Notably, this strategy does not add any additional weights or computational overhead to the original representation during inference. Extensive experiments demonstrate that our method produces high-quality colorized views for indoor and outdoor scenes, showcasing significant cross-view consistency advantages over baseline approaches. Our method is agnostic to the underlying 3D representation and easily generalizable to NeRF and 3DGS methods. Further, we validate the efficacy of our approach in several diverse applications: 1.) Infra-Red (IR) multi-view images and 2.) Legacy grayscale multi-view image sequences. Project Webpage: https://val.cds.iisc.ac.in/chroma-distill.github.io/