Collaborative Augmented Reality on Smartphones via Life-long City-scale Maps

TL;DR

This paper introduces a comprehensive city-scale augmented reality system on smartphones, featuring a new experience-based mapping framework, real-time multi-device localization, and deployment in San Francisco, supported by a large public dataset.

Contribution

It presents the first end-to-end production AR system for city-scale environments on mobile devices, with novel experience-based mapping and synchronization techniques.

Findings

System successfully deployed in San Francisco covering hundreds of kilometers

Achieves real-time multi-device localization despite network limitations

Provides the largest public dataset for city-scale AR mapping

Abstract

In this paper we present the first published end-to-end production computer-vision system for powering city-scale shared augmented reality experiences on mobile devices. In doing so we propose a new formulation for an experience-based mapping framework as an effective solution to the key issues of city-scale SLAM scalability, robustness, map updates and all-time all-weather performance required by a production system. Furthermore, we propose an effective way of synchronising SLAM systems to deliver seamless real-time localisation of multiple edge devices at the same time. All this in the presence of network latency and bandwidth limitations. The resulting system is deployed and tested at scale in San Francisco where it delivers AR experiences in a mapped area of several hundred kilometers. To foster further development of this area we offer the data set to the public, constituting the largest of this kind to date.