Interpretable Semantic Photo Geolocation

TL;DR

This paper introduces a semantic partitioning method and an importance metric to enhance the interpretability of deep learning models for photo geolocalization, achieving high accuracy and providing insights into model decisions.

Contribution

It presents a novel semantic partitioning approach and an importance metric, improving interpretability without sacrificing state-of-the-art accuracy in geolocalization.

Findings

Achieved state-of-the-art geolocalization accuracy on benchmark datasets.

Proposed semantic partitioning improves understanding of model predictions.

Importance metric enables large-scale analysis of model decision factors.

Abstract

Planet-scale photo geolocalization is the complex task of estimating the location depicted in an image solely based on its visual content. Due to the success of convolutional neural networks (CNNs), current approaches achieve super-human performance. However, previous work has exclusively focused on optimizing geolocalization accuracy. Due to the black-box property of deep learning systems, their predictions are difficult to validate for humans. State-of-the-art methods treat the task as a classification problem, where the choice of the classes, that is the partitioning of the world map, is crucial for the performance. In this paper, we present two contributions to improve the interpretability of a geolocalization model: (1) We propose a novel semantic partitioning method which intuitively leads to an improved understanding of the predictions, while achieving state-of-the-art results for geolocational accuracy on benchmark test sets; (2) We introduce a metric to assess the importance of semantic visual concepts for a certain prediction to provide additional interpretable information, which allows for a large-scale analysis of already trained models. Source code and dataset are publicly available.