Parameter-Efficient Fine-Tuning of Multispectral Foundation Models for Hyperspectral Image Classification

TL;DR

This paper introduces an efficient parameter-efficient fine-tuning framework for hyperspectral image classification using SpectralGPT, achieving competitive results with minimal trainable parameters and storage, and outperforming some dedicated models.

Contribution

The paper proposes KronA+ and other PEFT methods for hyperspectral image classification, significantly reducing training resources while maintaining or improving performance.

Findings

KronA+ achieves similar performance with only 0.056% trainable parameters.

Full fine-tuning of SpectralGPT outperforms some dedicated hyperspectral models.

PEFT methods require substantially less storage and training epochs.

Abstract

Foundation models have achieved great success across diverse domains, including remote sensing (RS), thanks to their versatility and strong generalization abilities. However, most RS foundation models are designed for multispectral data, while hyperspectral imagery (HSI) - with its hundreds of spectral bands - remains less explored. Fine-tuning such models for downstream tasks is also challenging, often demanding considerable memory and storage. In this paper, we propose an efficient framework to fine-tune SpectralGPT, a multispectral foundation model, for hyperspectral image classification (HSIC). We explore several Parameter-Efficient Fine-Tuning (PEFT) methods, including Low-Rank Adaptation (LoRA), Kronecker-based adaptation (KronA), Low-Rank Kronecker (LoKr), and the recent LoRA+, which uses distinct learning rates for low-rank adapters scaled by a factor lambda. Inspired by LoRA+, we introduce KronA+, which applies a similar mechanism to the Kronecker matrices. We evaluate our approach on five datasets from different sensors, showing competitive performance with state-of-the-art HSI models. Our full fine-tuning (FFT) setup for SpectralGPT even outperforms a dedicated hyperspectral foundation model on some datasets while requiring only a quarter of the training epochs. Under the same number of epochs, KronA+ reaches similar performance with far fewer trainable parameters - just 0.056 percent - and adds only approximately 0.2 megabytes of storage, making it the most effective PEFT method tested.