OmniCity: Omnipotent City Understanding with Multi-level and Multi-view Images

TL;DR

OmniCity is a comprehensive dataset combining multi-view satellite, street-level panorama, and mono-view images with extensive annotations, enabling advanced city understanding tasks and benchmarking state-of-the-art models.

Contribution

The paper introduces OmniCity, a large-scale, multi-view city dataset with efficient annotation pipeline and new benchmarks for diverse city understanding tasks.

Findings

OmniCity contains over 100K annotated images from 25K locations.

Provides benchmarks for building extraction, height estimation, and segmentation.

Introduces a new task for fine-grained building instance segmentation.

Abstract

This paper presents OmniCity, a new dataset for omnipotent city understanding from multi-level and multi-view images. More precisely, the OmniCity contains multi-view satellite images as well as street-level panorama and mono-view images, constituting over 100K pixel-wise annotated images that are well-aligned and collected from 25K geo-locations in New York City. To alleviate the substantial pixel-wise annotation efforts, we propose an efficient street-view image annotation pipeline that leverages the existing label maps of satellite view and the transformation relations between different views (satellite, panorama, and mono-view). With the new OmniCity dataset, we provide benchmarks for a variety of tasks including building footprint extraction, height estimation, and building plane/instance/fine-grained segmentation. Compared with the existing multi-level and multi-view benchmarks, OmniCity contains a larger number of images with richer annotation types and more views, provides more benchmark results of state-of-the-art models, and introduces a novel task for fine-grained building instance segmentation on street-level panorama images. Moreover, OmniCity provides new problem settings for existing tasks, such as cross-view image matching, synthesis, segmentation, detection, etc., and facilitates the developing of new methods for large-scale city understanding, reconstruction, and simulation. The OmniCity dataset as well as the benchmarks will be available at https://city-super.github.io/omnicity.