Interpretable Perception and Reasoning for Audiovisual Geolocation

TL;DR

This paper introduces a novel audiovisual geolocation framework that combines interpretable perception and reasoning to improve global localization accuracy using multimodal cues, supported by a new large-scale video benchmark.

Contribution

It presents AVG, a comprehensive audiovisual geolocation benchmark, and a three-stage framework integrating perception, reasoning, and prediction for enhanced localization.

Findings

Framework significantly outperforms unimodal baselines.

Interpretable soundscape perception provides critical localization signals.

Multimodal reasoning with audio and visual cues achieves high-precision geolocation.

Abstract

While recent advances in Multimodal Large Language Models (MLLMs) have improved image-based localization, precise global geolocation remains a formidable challenge due to the inherent ambiguity of visual landscapes and the largely untapped potential of auditory cues. In this paper, we introduce Audiovisual Geolocation, a framework designed to resolve geographic ambiguity through interpretable perception and reasoning. We present AVG, a high-quality global-scale video benchmark for geolocation, comprising 20,000 curated clips across 1,000 distinct locations. To address the complexity of audiovisual geolocation, we propose a three-stage framework: (1) a Perception stage that utilizes a mixture-autoregressive sparse autoencoder to decompose noisy audio into semantically grounded "acoustic atoms"; (2) a Multimodal Reasoning stage that employs an MLLM finetuned via Group Relative Policy Optimization (GRPO) to synthesize these atoms with visual features; and (3) a Precision Prediction stage using Riemannian Flow Matching on the $S^2$ manifold. Our experiments demonstrate that our framework significantly outperforms unimodal baselines. These results entail that interpretable perception of the soundscape provides a critical, orthogonal signal that, when coupled with multimodal reasoning, enables high-precision global localization.