FUN: A Focal U-Net Combining Reconstruction and Object Detection for Snapshot Spectral Imaging

TL;DR

The paper introduces FUN, an end-to-end neural network that jointly reconstructs hyperspectral images and detects objects in real-time, using a novel focal modulation mechanism to reduce computational complexity.

Contribution

It proposes a new multi-task learning framework with a shared U-shaped backbone and focal modulation, enabling efficient joint hyperspectral image reconstruction and object detection.

Findings

Achieves state-of-the-art performance on reconstruction and detection tasks.

Uses 40% fewer parameters and 30% less computation than recent methods.

Provides a new annotated dataset for hyperspectral object detection.

Abstract

Conventional push-broom hyperspectral imaging suffers from slow acquisition speeds, precluding real-time object detection; in contrast, snapshot spectral imaging enables instantaneous hyperspectral images (HSIs) capture, making real-time object detection feasible, yet its potential is often compromised by time-consuming post-capture reconstruction. To address this issue, we propose the Focal U-shaped Network (FUN), a novel end-to-end framework that jointly performs HSI reconstruction and object detection via multi-task learning. FUN employs a shared U-shaped backbone, where reconstruction provides underlying spectral information while detection guides semantic-aware priors learning, facilitating mutually beneficial task interaction. Crucially, we introduce focal modulation, an efficient alternative to self-attention that modulates spatial and spectral features while reducing quadratic computational complexity, enabling a self-attention-free architecture for joint reconstruction and detection. Furthermore, we contribute a new HSI object detection dataset with 8712 annotated objects across 363 HSIs to facilitate evaluation of the proposed method. Experiments demonstrate that FUN achieves state-of-the-art performance on both tasks, using 40% fewer parameters and 30% less computation than recent alternatives, making it promising for future real-time edge deployment. The code and datasets are available: https://github.com/ShawnDong98/FUN.