SpecDM: Hyperspectral Dataset Synthesis with Pixel-level Semantic Annotations

TL;DR

This paper introduces SpecDM, a novel diffusion-based method for synthesizing high-dimensional hyperspectral images with pixel-level annotations, aiding in training data generation for semantic segmentation and change detection tasks.

Contribution

It is the first to generate annotated hyperspectral datasets using a diffusion model with a two-stream VAE architecture.

Findings

Synthetic datasets improve downstream task performance.

First successful generation of high-dimensional hyperspectral images with annotations.

Demonstrates the effectiveness of diffusion models in hyperspectral data synthesis.

Abstract

In hyperspectral remote sensing field, some downstream dense prediction tasks, such as semantic segmentation (SS) and change detection (CD), rely on supervised learning to improve model performance and require a large amount of manually annotated data for training. However, due to the needs of specific equipment and special application scenarios, the acquisition and annotation of hyperspectral images (HSIs) are often costly and time-consuming. To this end, our work explores the potential of generative diffusion model in synthesizing HSIs with pixel-level annotations. The main idea is to utilize a two-stream VAE to learn the latent representations of images and corresponding masks respectively, learn their joint distribution during the diffusion model training, and finally obtain the image and mask through their respective decoders. To the best of our knowledge, it is the first work to generate high-dimensional HSIs with annotations. Our proposed approach can be applied in various kinds of dataset generation. We select two of the most widely used dense prediction tasks: semantic segmentation and change detection, and generate datasets suitable for these tasks. Experiments demonstrate that our synthetic datasets have a positive impact on the improvement of these downstream tasks.