A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarking

TL;DR

This paper conducts the first comprehensive survey and benchmarking of the robustness of deep neural networks in remote sensing image classification and object detection, highlighting vulnerabilities and guiding future robustness improvements.

Contribution

It introduces curated datasets with natural and adversarial noises and provides a thorough evaluation of various models' robustness in remote sensing tasks.

Findings

Models are vulnerable to adversarial noises in remote sensing

Training methods influence model robustness to adversarial attacks

Benchmark datasets enable standardized robustness evaluation

Abstract

Deep neural networks (DNNs) have found widespread applications in interpreting remote sensing (RS) imagery. However, it has been demonstrated in previous works that DNNs are vulnerable to different types of noises, particularly adversarial noises. Surprisingly, there has been a lack of comprehensive studies on the robustness of RS tasks, prompting us to undertake a thorough survey and benchmark on the robustness of image classification and object detection in RS. To our best knowledge, this study represents the first comprehensive examination of both natural robustness and adversarial robustness in RS tasks. Specifically, we have curated and made publicly available datasets that contain natural and adversarial noises. These datasets serve as valuable resources for evaluating the robustness of DNNs-based models. To provide a comprehensive assessment of model robustness, we conducted meticulous experiments with numerous different classifiers and detectors, encompassing a wide range of mainstream methods. Through rigorous evaluation, we have uncovered insightful and intriguing findings, which shed light on the relationship between adversarial noise crafting and model training, yielding a deeper understanding of the susceptibility and limitations of various models, and providing guidance for the development of more resilient and robust models