Multi-Spectral Gaussian Splatting with Neural Color Representation

TL;DR

This paper introduces MS-Splatting, a multi-spectral 3D Gaussian Splatting framework that generates consistent multi-view images across various spectral domains without calibration, using a neural color representation for improved rendering quality.

Contribution

The paper proposes a novel neural color encoding for multi-spectral 3D Gaussian Splatting that jointly models spectral data, enhancing rendering quality and versatility across different spectral modalities.

Findings

Improved multi-spectral rendering quality over state-of-the-art methods.

Effective joint learning of multiple spectral bands within a unified representation.

Successful application in agricultural imaging, such as NDVI rendering.

Abstract

We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting (3DGS) framework that is able to generate multi-view consistent novel views from images of multiple, independent cameras with different spectral domains. In contrast to previous approaches, our method does not require cross-modal camera calibration and is versatile enough to model a variety of different spectra, including thermal and near-infra red, without any algorithmic changes. Unlike existing 3DGS-based frameworks that treat each modality separately (by optimizing per-channel spherical harmonics) and therefore fail to exploit the underlying spectral and spatial correlations, our method leverages a novel neural color representation that encodes multi-spectral information into a learned, compact, per-splat feature embedding. A shallow multi-layer perceptron (MLP) then decodes this embedding to obtain spectral color values, enabling joint learning of all bands within a unified representation. Our experiments show that this simple yet effective strategy is able to improve multi-spectral rendering quality, while also leading to improved per-spectra rendering quality over state-of-the-art methods. We demonstrate the effectiveness of this new technique in agricultural applications to render vegetation indices, such as normalized difference vegetation index (NDVI).