SeMAnD: Self-Supervised Anomaly Detection in Multimodal Geospatial Datasets

TL;DR

SeMAnD is a self-supervised method for detecting geometric anomalies in multimodal geospatial datasets, leveraging data augmentation and discriminative representation learning to identify small local defects across diverse regions.

Contribution

The paper introduces SeMAnD, a novel self-supervised approach with RandPolyAugment for multimodal geospatial anomaly detection, outperforming existing methods and demonstrating scalability with multiple data modalities.

Findings

SeMAnD outperforms domain-agnostic strategies by 4.8-19.7% in anomaly detection AUC.

Model performance improves by up to 20.4% with more input modalities.

Increased diversity and strength of data augmentations boost performance by up to 22.9%.

Abstract

We propose a Self-supervised Anomaly Detection technique, called SeMAnD, to detect geometric anomalies in Multimodal geospatial datasets. Geospatial data comprises of acquired and derived heterogeneous data modalities that we transform to semantically meaningful, image-like tensors to address the challenges of representation, alignment, and fusion of multimodal data. SeMAnD is comprised of (i) a simple data augmentation strategy, called RandPolyAugment, capable of generating diverse augmentations of vector geometries, and (ii) a self-supervised training objective with three components that incentivize learning representations of multimodal data that are discriminative to local changes in one modality which are not corroborated by the other modalities. Detecting local defects is crucial for geospatial anomaly detection where even small anomalies (e.g., shifted, incorrectly connected, malformed, or missing polygonal vector geometries like roads, buildings, landcover, etc.) are detrimental to the experience and safety of users of geospatial applications like mapping, routing, search, and recommendation systems. Our empirical study on test sets of different types of real-world geometric geospatial anomalies across 3 diverse geographical regions demonstrates that SeMAnD is able to detect real-world defects and outperforms domain-agnostic anomaly detection strategies by 4.8-19.7% as measured using anomaly classification AUC. We also show that model performance increases (i) up to 20.4% as the number of input modalities increase and (ii) up to 22.9% as the diversity and strength of training data augmentations increase.