PlaNet - Photo Geolocation with Convolutional Neural Networks

TL;DR

This paper introduces PlaNet, a deep learning system that classifies images into geographic regions to determine their location, outperforming previous methods and achieving superhuman accuracy by leveraging multi-scale cues and temporal coherence.

Contribution

The paper presents a novel classification-based approach to photo geolocation using deep CNNs trained on millions of images, surpassing prior landmark recognition methods.

Findings

PlaNet outperforms previous geolocation approaches.

The model achieves superhuman accuracy in some cases.

Combining PlaNet with LSTM improves performance by 50%.

Abstract

Is it possible to build a system to determine the location where a photo was taken using just its pixels? In general, the problem seems exceptionally difficult: it is trivial to construct situations where no location can be inferred. Yet images often contain informative cues such as landmarks, weather patterns, vegetation, road markings, and architectural details, which in combination may allow one to determine an approximate location and occasionally an exact location. Websites such as GeoGuessr and View from your Window suggest that humans are relatively good at integrating these cues to geolocate images, especially en-masse. In computer vision, the photo geolocation problem is usually approached using image retrieval methods. In contrast, we pose the problem as one of classification by subdividing the surface of the earth into thousands of multi-scale geographic cells, and train a deep network using millions of geotagged images. While previous approaches only recognize landmarks or perform approximate matching using global image descriptors, our model is able to use and integrate multiple visible cues. We show that the resulting model, called PlaNet, outperforms previous approaches and even attains superhuman levels of accuracy in some cases. Moreover, we extend our model to photo albums by combining it with a long short-term memory (LSTM) architecture. By learning to exploit temporal coherence to geolocate uncertain photos, we demonstrate that this model achieves a 50% performance improvement over the single-image model.