DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications

TL;DR

DUNIA introduces pixel-sized embeddings learned through cross-modal alignment of images and LiDAR data, enabling effective zero-shot and fine-tuned environmental monitoring across diverse tasks.

Contribution

The paper presents a novel method for learning pixel-level embeddings via cross-modal contrastive learning, improving integration and performance in Earth observation applications.

Findings

Effective zero-shot performance on multiple environmental tasks.

Outperforms specialized models in low-data regimes.

Achieves state-of-the-art results in fine-tuning on several tasks.

Abstract

Significant efforts have been directed towards adapting self-supervised multimodal learning for Earth observation applications. However, most current methods produce coarse patch-sized embeddings, limiting their effectiveness and integration with other modalities like LiDAR. To close this gap, we present DUNIA, an approach to learn pixel-sized embeddings through cross-modal alignment between images and full-waveform LiDAR data. As the model is trained in a contrastive manner, the embeddings can be directly leveraged in the context of a variety of environmental monitoring tasks in a zero-shot setting. In our experiments, we demonstrate the effectiveness of the embeddings for seven such tasks: canopy height mapping, fractional canopy cover, land cover mapping, tree species identification, plant area index, crop type classification, and per-pixel waveform-based vertical structure mapping. The results show that the embeddings, along with zero-shot classifiers, often outperform specialized supervised models, even in low-data regimes. In the fine-tuning setting, we show strong performances near or better than the state-of-the-art on five out of six tasks.