Lifting GIS Maps into Strong Geometric Context for Scene Understanding

TL;DR

This paper introduces a method to incorporate GIS map data into scene understanding by generating 3D geometric priors from 2D GIS information and unorganized photos, improving depth, segmentation, and detection accuracy.

Contribution

It presents a pipeline to create accurate 3D geometric priors from GIS data using SfM, enhancing various visual tasks with minimal user input.

Findings

Geometric predictions outperform single-image depth methods.

GIS context improves pedestrian detection re-scoring.

Enhanced semantic segmentation accuracy with GIS features.

Abstract

Contextual information can have a substantial impact on the performance of visual tasks such as semantic segmentation, object detection, and geometric estimation. Data stored in Geographic Information Systems (GIS) offers a rich source of contextual information that has been largely untapped by computer vision. We propose to leverage such information for scene understanding by combining GIS resources with large sets of unorganized photographs using Structure from Motion (SfM) techniques. We present a pipeline to quickly generate strong 3D geometric priors from 2D GIS data using SfM models aligned with minimal user input. Given an image resectioned against this model, we generate robust predictions of depth, surface normals, and semantic labels. We show that the precision of the predicted geometry is substantially more accurate other single-image depth estimation methods. We then demonstrate the utility of these contextual constraints for re-scoring pedestrian detections, and use these GIS contextual features alongside object detection score maps to improve a CRF-based semantic segmentation framework, boosting accuracy over baseline models.