Using Multiple Input Modalities Can Improve Data-Efficiency and O.O.D. Generalization for ML with Satellite Imagery

TL;DR

This paper demonstrates that incorporating multiple geographic data modalities with satellite imagery enhances machine learning performance, especially in data-limited and out-of-sample scenarios, with simple fusion methods outperforming learned ones.

Contribution

It introduces augmented datasets combining satellite imagery with other geographic data layers and shows that simple fusion strategies improve model performance over traditional optical-only approaches.

Findings

Multi-modal inputs improve model accuracy.

Simple fusion strategies outperform learned methods.

Performance gains are largest with limited labeled data.

Abstract

A large variety of geospatial data layers is available around the world ranging from remotely-sensed raster data like satellite imagery, digital elevation models, predicted land cover maps, and human-annotated data, to data derived from environmental sensors such as air temperature or wind speed data. A large majority of machine learning models trained on satellite imagery (SatML), however, are designed primarily for optical input modalities such as multi-spectral satellite imagery. To better understand the value of using other input modalities alongside optical imagery in supervised learning settings, we generate augmented versions of SatML benchmark tasks by appending additional geographic data layers to datasets spanning classification, regression, and segmentation. Using these augmented datasets, we find that fusing additional geographic inputs with optical imagery can significantly improve SatML model performance. Benefits are largest in settings where labeled data are limited and in geographic out-of-sample settings, suggesting that multi-modal inputs may be especially valuable for data-efficiency and out-of-sample performance of SatML models. Surprisingly, we find that hard-coded fusion strategies outperform learned variants, with interesting implications for future work.