A Vision-Language Framework for Multispectral Scene Representation Using Language-Grounded Features

TL;DR

This paper introduces Spectral LLaVA, a novel vision-language framework that integrates multispectral remote sensing data with language grounding to improve scene understanding and description accuracy in complex environments.

Contribution

The paper presents a new multispectral vision-language framework that enhances scene representation and description in remote sensing, leveraging a lightweight alignment layer and frozen vision backbone.

Findings

Improved scene description accuracy with multispectral data

Enhanced classification performance through semantic feature refinement

Effective alignment of multispectral features with language models

Abstract

Scene understanding in remote sensing often faces challenges in generating accurate representations for complex environments such as various land use areas or coastal regions, which may also include snow, clouds, or haze. To address this, we present a vision-language framework named Spectral LLaVA, which integrates multispectral data with vision-language alignment techniques to enhance scene representation and description. Using the BigEarthNet v2 dataset from Sentinel-2, we establish a baseline with RGB-based scene descriptions and further demonstrate substantial improvements through the incorporation of multispectral information. Our framework optimizes a lightweight linear projection layer for alignment while keeping the vision backbone of SpectralGPT frozen. Our experiments encompass scene classification using linear probing and language modeling for jointly performing scene classification and description generation. Our results highlight Spectral LLaVA's ability to produce detailed and accurate descriptions, particularly for scenarios where RGB data alone proves inadequate, while also enhancing classification performance by refining SpectralGPT features into semantically meaningful representations.