Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations

TL;DR

This paper presents a method to automatically generate large-scale vision-language datasets in remote sensing using machine learning, enabling the development of more effective foundation models without human annotations.

Contribution

The study introduces an automated dataset curation approach for remote sensing vision-language models, significantly increasing dataset size and improving downstream task performance.

Findings

Achieved 9.6 million vision-language pairs in VHR imagery.

Model outperforms baselines in zero-shot classification and retrieval.

Superior results in vision-only tasks like linear probing and k-NN.

Abstract

The prominence of generalized foundation models in vision-language integration has witnessed a surge, given their multifarious applications. Within the natural domain, the procurement of vision-language datasets to construct these foundation models is facilitated by their abundant availability and the ease of web crawling. Conversely, in the remote sensing domain, although vision-language datasets exist, their volume is suboptimal for constructing robust foundation models. This study introduces an approach to curate vision-language datasets by employing an image decoding machine learning model, negating the need for human-annotated labels. Utilizing this methodology, we amassed approximately 9.6 million vision-language paired datasets in VHR imagery. The resultant model outperformed counterparts that did not leverage publicly available vision-language datasets, particularly in downstream tasks such as zero-shot classification, semantic localization, and image-text retrieval. Moreover, in tasks exclusively employing vision encoders, such as linear probing and k-NN classification, our model demonstrated superior efficacy compared to those relying on domain-specific vision-language datasets.