Zero-Shot Satellite Image Retrieval through Joint Embeddings: Application to Crisis Response

TL;DR

GeoQuery enables zero-shot satellite image retrieval using joint embeddings, facilitating crisis response by matching natural language queries with global Earth observation data without extensive training.

Contribution

The paper introduces a novel two-stage semantic and visual search system that leverages prompt-aligned proxies to enable zero-shot retrieval of satellite imagery for crisis response.

Findings

Achieved 31.6% accuracy within 50 km on disaster-location queries.

Strong performance on flood-related queries with 50% accuracy within 50 km.

Deployed in a crisis system, successfully identified vulnerable areas during Cyclone Alfred.

Abstract

Semantic search of Earth observation archives remains challenging. Visual foundation models such as CLAY produce rich embeddings of satellite imagery but lack the natural-language grounding needed for intuitive query, and full contrastive training of a remote-sensing CLIP-style model requires paired data and compute that are unavailable at global scale. To allow natural language querying at global scales, we present GeoQuery, a zero-shot retrieval system that sidesteps data and compute constraints through a two-stage semantic and visual search, leveraging a natural language embedding of a subset (proxy) of global data. Rather than training a joint encoder, we generate language descriptions for a 100k proxy subset of global Sentinel-2 tiles and optimise the description-generation prompt so that distances in the resulting text-embedding space correlate with distances in the frozen CLAY visual-embedding space. Queries are resolved in two stages, with a text-similarity search over the proxy subset followed by a visual nearest-neighbour search over worldwide CLAY embeddings On 76 disaster-location queries covering UK floods, US wildfires, and US droughts, GeoQuery achieves 31.6\% accuracy within 50\,km, with the strongest performance on floods (50\% within 50\,km) where terrain features are well captured by RGB embeddings. Deployed within a crisis response system called \ECHO{}, GeoQuery identified vulnerable areas during Brisbane's 2025 Cyclone Alfred, with downstream flood simulations reproducing historical patterns. Prompt-aligned proxies offer a practical bridge between EO foundation models and operational retrieval when full contrastive training is out of reach.