Multiple Distribution Shift -- Aerial (MDS-A): A Dataset for Test-Time Error Detection and Model Adaptation

TL;DR

This paper introduces MDS-A, a comprehensive aerial dataset with simulated weather-induced distribution shifts, to evaluate and improve model robustness and error detection in out-of-distribution scenarios.

Contribution

The paper presents MDS-A, a novel dataset with diverse weather conditions for aerial imagery, and evaluates baseline models and error detection techniques under distribution shifts.

Findings

Baseline models' performance degrades under weather-induced shifts.

Error detection techniques improve out-of-distribution performance.

MDS-A enables systematic evaluation of model robustness.

Abstract

Machine learning models assume that training and test samples are drawn from the same distribution. As such, significant differences between training and test distributions often lead to degradations in performance. We introduce Multiple Distribution Shift -- Aerial (MDS-A) -- a collection of inter-related datasets of the same aerial domain that are perturbed in different ways to better characterize the effects of out-of-distribution performance. Specifically, MDS-A is a set of simulated aerial datasets collected under different weather conditions. We include six datasets under different simulated weather conditions along with six baseline object-detection models, as well as several test datasets that are a mix of weather conditions that we show have significant differences from the training data. In this paper, we present characterizations of MDS-A, provide performance results for the baseline machine learning models (on both their specific training datasets and the test data), as well as results of the baselines after employing recent knowledge-engineering error-detection techniques (EDR) thought to improve out-of-distribution performance. The dataset is available at https://lab-v2.github.io/mdsa-dataset-website.