Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mapping

TL;DR

Sat2Sound introduces a multimodal framework that leverages satellite images, audio, and text descriptions to improve geospatial soundscape mapping and enable soundscape synthesis.

Contribution

It combines vision, language, and audio data with contrastive learning to discover shared soundscape concepts and achieve state-of-the-art cross-modal retrieval performance.

Findings

Achieves state-of-the-art results on GeoSound and SoundingEarth benchmarks.

Enables detailed soundscape caption retrieval for synthesis and educational use.

Demonstrates effective cross-modal mapping between satellite images and sound descriptions.

Abstract

We present Sat2Sound, a unified multimodal framework for geospatial soundscape understanding, designed to predict and map the distribution of sounds across the Earth's surface. Existing methods for this task rely on paired satellite images and geotagged audio samples, which often fail to capture the full diversity of sound at a location. Sat2Sound overcomes this limitation by augmenting datasets with semantically rich, vision-language model-generated soundscape descriptions, which broaden the range of possible ambient sounds represented at each location. Our framework jointly learns from audio, text descriptions of audio, satellite images, and synthetic image captions through contrastive and codebook-aligned learning, discovering a set of "soundscape concepts" shared across modalities, enabling hyper-localized, explainable soundscape mapping. Sat2Sound achieves state-of-the-art performance in cross-modal retrieval between satellite image and audio on the GeoSound and SoundingEarth benchmarks. Finally, by retrieving detailed soundscape captions that can be rendered through text-to-audio models, Sat2Sound enables location-conditioned soundscape synthesis for immersive and educational applications, even with limited computational resources. Our code and models are available at https://github.com/mvrl/sat2sound.