Semantic Image Alignment for Vehicle Localization

TL;DR

This paper introduces a novel monocular camera-based vehicle localization method that aligns semantic images with dense semantic maps, enabling accurate, real-time localization without relying on keypoints or LiDAR.

Contribution

The approach formulates localization as direct image alignment on semantic images, eliminating the need for handcrafted features or expensive sensors, and works across various dense semantic map types.

Findings

Achieves reliable, real-time vehicle localization in diverse semantic maps.

Does not require keypoints, handcrafted landmarks, or LiDAR sensors.

Demonstrates broad applicability across stereo, LiDAR, and manually annotated maps.

Abstract

Accurate and reliable localization is a fundamental requirement for autonomous vehicles to use map information in higher-level tasks such as navigation or planning. In this paper, we present a novel approach to vehicle localization in dense semantic maps, including vectorized high-definition maps or 3D meshes, using semantic segmentation from a monocular camera. We formulate the localization task as a direct image alignment problem on semantic images, which allows our approach to robustly track the vehicle pose in semantically labeled maps by aligning virtual camera views rendered from the map to sequences of semantically segmented camera images. In contrast to existing visual localization approaches, the system does not require additional keypoint features, handcrafted localization landmark extractors or expensive LiDAR sensors. We demonstrate the wide applicability of our method on a diverse set of semantic mesh maps generated from stereo or LiDAR as well as manually annotated HD maps and show that it achieves reliable and accurate localization in real-time.