Online Sensor Hallucination via Knowledge Distillation for Multimodal Image Classification

TL;DR

This paper introduces a knowledge distillation-based hallucination architecture that enables satellite image classifiers to approximate missing sensor modalities during testing, improving classification performance when some sensor data is unavailable.

Contribution

It proposes a novel modular hallucination framework using knowledge distillation to handle missing modalities in multimodal satellite image classification.

Findings

Effective in approximating missing sensor data during testing.

Improves scene classification accuracy with incomplete sensor information.

Validated on large-scale PAN-MS and hyperspectral datasets.

Abstract

We deal with the problem of information fusion driven satellite image/scene classification and propose a generic hallucination architecture considering that all the available sensor information are present during training while some of the image modalities may be absent while testing. It is well-known that different sensors are capable of capturing complementary information for a given geographical area and a classification module incorporating information from all the sources are expected to produce an improved performance as compared to considering only a subset of the modalities. However, the classical classifier systems inherently require all the features used to train the module to be present for the test instances as well, which may not always be possible for typical remote sensing applications (say, disaster management). As a remedy, we provide a robust solution in terms of a hallucination module that can approximate the missing modalities from the available ones during the decision-making stage. In order to ensure better knowledge transfer during modality hallucination, we explicitly incorporate concepts of knowledge distillation for the purpose of exploring the privileged (side) information in our framework and subsequently introduce an intuitive modular training approach. The proposed network is evaluated extensively on a large-scale corpus of PAN-MS image pairs (scene recognition) as well as on a benchmark hyperspectral image dataset (image classification) where we follow different experimental scenarios and find that the proposed hallucination based module indeed is capable of capturing the multi-source information, albeit the explicit absence of some of the sensor information, and aid in improved scene characterization.