Enhancing Ground-to-Aerial Image Matching for Visual Misinformation Detection Using Semantic Segmentation

TL;DR

This paper introduces SAN-QUAD, a novel architecture that improves ground-to-aerial image matching for geolocation tasks by incorporating semantic segmentation, addressing challenges posed by varying fields of view and untagged images.

Contribution

The study presents SAN-QUAD, a four-stream Siamese-like network that leverages semantic segmentation to enhance image matching accuracy without external geolocation data.

Findings

Achieved up to 9.8% improvement over previous methods.

Effectively handles varying fields of view in image matching.

Demonstrated robustness on the CVUSA dataset.

Abstract

The recent advancements in generative AI techniques, which have significantly increased the online dissemination of altered images and videos, have raised serious concerns about the credibility of digital media available on the Internet and distributed through information channels and social networks. This issue particularly affects domains that rely heavily on trustworthy data, such as journalism, forensic analysis, and Earth observation. To address these concerns, the ability to geolocate a non-geo-tagged ground-view image without external information, such as GPS coordinates, has become increasingly critical. This study tackles the challenge of linking a ground-view image, potentially exhibiting varying fields of view (FoV), to its corresponding satellite image without the aid of GPS data. To achieve this, we propose a novel four-stream Siamese-like architecture, the Quadruple Semantic Align Net (SAN-QUAD), which extends previous state-of-the-art (SOTA) approaches by leveraging semantic segmentation applied to both ground and satellite imagery. Experimental results on a subset of the CVUSA dataset demonstrate significant improvements of up to 9.8% over prior methods across various FoV settings.