CroCo: Cross-Modal Contrastive learning for localization of Earth Observation data

TL;DR

This paper introduces CroCo, a contrastive learning approach that effectively localizes aerial LiDAR-derived DEMs on optical imagery, achieving promising accuracy and demonstrating potential for broader applications in Earth observation data localization.

Contribution

The paper presents a novel contrastive learning framework for aligning DEM and optical imagery for localization, with extensive experiments on data sampling and hyperparameters.

Findings

Top-1 score of 0.71 achieved

Top-5 score of 0.81 achieved

Method shows promise for feature learning from RGB and DEM

Abstract

It is of interest to localize a ground-based LiDAR point cloud on remote sensing imagery. In this work, we tackle a subtask of this problem, i.e. to map a digital elevation model (DEM) rasterized from aerial LiDAR point cloud on the aerial imagery. We proposed a contrastive learning-based method that trains on DEM and high-resolution optical imagery and experiment the framework on different data sampling strategies and hyperparameters. In the best scenario, the Top-1 score of 0.71 and Top-5 score of 0.81 are obtained. The proposed method is promising for feature learning from RGB and DEM for localization and is potentially applicable to other data sources too. Source code will be released at https://github.com/wtseng530/AVLocalization.