Predicting Ground-Level Scene Layout from Aerial Imagery

TL;DR

This paper presents a novel end-to-end method for extracting semantic features from aerial images by predicting from ground imagery, enabling improved semantic segmentation, geolocation, and panorama generation without manual labels.

Contribution

The authors introduce a new strategy that predicts semantic features from aerial images using co-located ground imagery, eliminating manual labeling and enabling multiple downstream tasks.

Findings

Model can perform rough semantic labeling of aerial images.

Fine-tuning improves segmentation accuracy over baseline strategies.

Features enable geolocation, orientation estimation, and panorama hallucination.

Abstract

We introduce a novel strategy for learning to extract semantically meaningful features from aerial imagery. Instead of manually labeling the aerial imagery, we propose to predict (noisy) semantic features automatically extracted from co-located ground imagery. Our network architecture takes an aerial image as input, extracts features using a convolutional neural network, and then applies an adaptive transformation to map these features into the ground-level perspective. We use an end-to-end learning approach to minimize the difference between the semantic segmentation extracted directly from the ground image and the semantic segmentation predicted solely based on the aerial image. We show that a model learned using this strategy, with no additional training, is already capable of rough semantic labeling of aerial imagery. Furthermore, we demonstrate that by finetuning this model we can achieve more accurate semantic segmentation than two baseline initialization strategies. We use our network to address the task of estimating the geolocation and geoorientation of a ground image. Finally, we show how features extracted from an aerial image can be used to hallucinate a plausible ground-level panorama.