Prototype-Based Information Compensation Network for Multi-Source Remote Sensing Data Classification

TL;DR

This paper introduces PICNet, a novel neural network that enhances multi-source remote sensing data classification by improving inter-frequency feature coupling and modeling global complementary information through prototype-based modules.

Contribution

The paper proposes a prototype-based information compensation network with a frequency interaction module for better multi-source feature integration in remote sensing.

Findings

PICNet outperforms state-of-the-art methods on three public datasets.

The frequency interaction module enhances inter-frequency feature coupling.

Prototype-based modules effectively model global multi-source complementary information.

Abstract

Multi-source remote sensing data joint classification aims to provide accuracy and reliability of land cover classification by leveraging the complementary information from multiple data sources. Existing methods confront two challenges: inter-frequency multi-source feature coupling and inconsistency of complementary information exploration. To solve these issues, we present a Prototype-based Information Compensation Network (PICNet) for land cover classification based on HSI and SAR/LiDAR data. Specifically, we first design a frequency interaction module to enhance the inter-frequency coupling in multi-source feature extraction. The multi-source features are first decoupled into high- and low-frequency components. Then, these features are recoupled to achieve efficient inter-frequency communication. Afterward, we design a prototype-based information compensation module to model the global multi-source complementary information. Two sets of learnable modality prototypes are introduced to represent the global modality information of multi-source data. Subsequently, cross-modal feature integration and alignment are achieved through cross-attention computation between the modality-specific prototype vectors and the raw feature representations. Extensive experiments on three public datasets demonstrate the significant superiority of our PICNet over state-of-the-art methods. The codes are available at https://github.com/oucailab/PICNet.