Visual Localization using Imperfect 3D Models from the Internet

TL;DR

This paper investigates the use of readily available, imperfect 3D models from the Internet for visual localization, creating a benchmark to evaluate their effectiveness and highlighting the potential and challenges of this approach.

Contribution

It introduces a new benchmark for evaluating Internet-derived 3D models in visual localization and provides a comprehensive experimental analysis of their impact on localization accuracy.

Findings

Internet 3D models are promising for quick scene representation.

Significant improvements are needed in localization pipelines for better accuracy.

The benchmark facilitates future research in this area.

Abstract

Visual localization is a core component in many applications, including augmented reality (AR). Localization algorithms compute the camera pose of a query image w.r.t. a scene representation, which is typically built from images. This often requires capturing and storing large amounts of data, followed by running Structure-from-Motion (SfM) algorithms. An interesting, and underexplored, source of data for building scene representations are 3D models that are readily available on the Internet, e.g., hand-drawn CAD models, 3D models generated from building footprints, or from aerial images. These models allow to perform visual localization right away without the time-consuming scene capturing and model building steps. Yet, it also comes with challenges as the available 3D models are often imperfect reflections of reality. E.g., the models might only have generic or no textures at all, might only provide a simple approximation of the scene geometry, or might be stretched. This paper studies how the imperfections of these models affect localization accuracy. We create a new benchmark for this task and provide a detailed experimental evaluation based on multiple 3D models per scene. We show that 3D models from the Internet show promise as an easy-to-obtain scene representation. At the same time, there is significant room for improvement for visual localization pipelines. To foster research on this interesting and challenging task, we release our benchmark at v-pnk.github.io/cadloc.