MCGA: Mixture of Codebooks Hyperspectral Reconstruction via Grayscale-Aware Attention

TL;DR

MCGA introduces a novel hyperspectral reconstruction method that explicitly leverages spectral priors and grayscale-aware attention, achieving state-of-the-art accuracy and efficiency in RGB-to-HSI tasks.

Contribution

The paper presents MCGA, a new framework combining mixture-of-codebooks and grayscale-aware attention to improve hyperspectral image reconstruction from RGB data.

Findings

Achieves state-of-the-art accuracy on multiple benchmarks.

Demonstrates 4-5x faster inference compared to existing methods.

Exhibits strong cross-dataset generalization.

Abstract

Reconstructing hyperspectral images (HSIs) from RGB inputs provides a cost-effective alternative to hyperspectral cameras, but reconstructing high-dimensional spectra from three channels is inherently ill-posed. Existing methods typically directly regress RGB-to-HSI mappings using large attention networks, which are computationally expensive and handle ill-posedness only implicitly. We propose MCGA, a Mixture-of-Codebooks with Grayscale-aware Attention framework that explicitly addresses these challenges using spectral priors and photometric consistency. MCGA first learns transferable spectral priors via a mixture-of-codebooks (MoC) from heterogeneous HSI datasets, then aligns RGB features with these priors through grayscale-aware photometric attention (GANet). Efficiency and robustness are further improved via top-K attention design and test-time adaptation (TTA). Experiments on multiple real-world benchmarks demonstrate the state-of-the-art accuracy, strong cross-dataset generalization, and 4-5x faster inference. Codes will be available once acceptance at https://github.com/Fibonaccirabbit/MCGA.