Squeezing nnU-Nets with Knowledge Distillation for On-Board Cloud Detection

TL;DR

This paper demonstrates that knowledge distillation can significantly compress nnU-Net models for satellite cloud detection, enabling efficient on-board processing without substantial performance loss.

Contribution

It introduces a method to compress large nnU-Net models into smaller U-Nets using knowledge distillation for on-board satellite cloud detection.

Findings

Achieved a Jaccard index of 0.882 on Sentinel-2 images.

Ranked within the top 7% in the Cloud Cover Detection Challenge.

Compressed models by approximately 280 times while maintaining performance.

Abstract

Cloud detection is a pivotal satellite image pre-processing step that can be performed both on the ground and on board a satellite to tag useful images. In the latter case, it can reduce the amount of data to downlink by pruning the cloudy areas, or to make a satellite more autonomous through data-driven acquisition re-scheduling. We approach this task with nnU-Nets, a self-reconfigurable framework able to perform meta-learning of a segmentation network over various datasets. Unfortunately, such models are commonly memory-inefficient due to their (very) large architectures. To benefit from them in on-board processing, we compress nnU-Nets with knowledge distillation into much smaller and compact U-Nets. Our experiments, performed over Sentinel-2 and Landsat-8 images revealed that nnU-Nets deliver state-of-the-art performance without any manual design. Our approach was ranked within the top 7% best solutions (across 847 teams) in the On Cloud N: Cloud Cover Detection Challenge, where we reached the Jaccard index of 0.882 over more than 10k unseen Sentinel-2 images (the winners obtained 0.897, the baseline U-Net with the ResNet-34 backbone: 0.817, and the classic Sentinel-2 image thresholding: 0.652). Finally, we showed that knowledge distillation enables to elaborate dramatically smaller (almost 280x) U-Nets when compared to nnU-Nets while still maintaining their segmentation capabilities.