Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth

TL;DR

PEFD is a novel framework that enables high-quality multispectral demosaicing from mosaiced measurements alone, leveraging projective geometry and pretrained models to outperform recent methods without ground truth.

Contribution

It introduces a perspective-equivariant fine-tuning approach that learns from measurements alone, utilizing geometric principles and pretrained models for improved multispectral reconstruction.

Findings

PEFD outperforms recent approaches on surgical and automotive datasets.

It recovers fine details like blood vessels and maintains spectral fidelity.

Demonstrates effectiveness on raw data from commercial multispectral sensors.

Abstract

Multispectral demosaicing is crucial to reconstruct full-resolution spectral images from snapshot mosaiced measurements, enabling real-time imaging from neurosurgery to autonomous driving. Classical methods are blurry, while supervised learning requires costly ground truth (GT) obtained from slow line-scanning systems. We propose Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), a framework that learns multispectral demosaicing from mosaiced measurements alone. PEFD a) exploits the projective geometry of camera-based imaging systems to leverage a richer group structure than previous demosaicing methods to recover more null-space information, and b) learns efficiently without GT by adapting pretrained foundation models designed for 1-3 channel imaging. On surgical and automotive datasets, PEFD recovers fine details such as blood vessels and preserves spectral fidelity, substantially outperforming recent approaches, nearing supervised performance. Furthermore, the performance of PEFD is demonstrated on raw, unprocessed data from a commercial multispectral sensor. Code is at https://github.com/Andrewwango/pefd.