Text2Pos: Text-to-Point-Cloud Cross-Modal Localization

TL;DR

This paper introduces Text2Pos, a novel cross-modal localization method that aligns natural language descriptions with point-cloud data to enable language-based navigation, supported by a new dataset and promising experimental results.

Contribution

We propose Text2Pos, a new cross-modal localization approach that aligns text with point clouds, and create KITTI360Pose, the first dataset for this task.

Findings

65% of queries localized within 15m in top-10 retrievals

First dataset for text-to-point-cloud localization based on KITTI360

Demonstrates potential for language-based navigation systems

Abstract

Natural language-based communication with mobile devices and home appliances is becoming increasingly popular and has the potential to become natural for communicating with mobile robots in the future. Towards this goal, we investigate cross-modal text-to-point-cloud localization that will allow us to specify, for example, a vehicle pick-up or goods delivery location. In particular, we propose Text2Pos, a cross-modal localization module that learns to align textual descriptions with localization cues in a coarse- to-fine manner. Given a point cloud of the environment, Text2Pos locates a position that is specified via a natural language-based description of the immediate surroundings. To train Text2Pos and study its performance, we construct KITTI360Pose, the first dataset for this task based on the recently introduced KITTI360 dataset. Our experiments show that we can localize 65% of textual queries within 15m distance to query locations for top-10 retrieved locations. This is a starting point that we hope will spark future developments towards language-based navigation.