Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language

TL;DR

Text2Loc++ is a novel neural network that effectively localizes 3D point cloud submaps using complex natural language descriptions, introducing new datasets and techniques for improved cross-modal alignment and robustness.

Contribution

The paper presents Text2Loc++, a new model with innovative training strategies and a city-scale dataset for language-guided 3D localization, advancing the state of the art.

Findings

Outperforms existing methods by up to 15% on KITTI360Pose

Demonstrates robust generalization to diverse urban scenes

Effectively handles complex linguistic expressions

Abstract

We tackle the problem of localizing 3D point cloud submaps using complex and diverse natural language descriptions, and present Text2Loc++, a novel neural network designed for effective cross-modal alignment between language and point clouds in a coarse-to-fine localization pipeline. To support benchmarking, we introduce a new city-scale dataset covering both color and non-color point clouds from diverse urban scenes, and organize location descriptions into three levels of linguistic complexity. In the global place recognition stage, Text2Loc++ combines a pretrained language model with a Hierarchical Transformer with Max pooling (HTM) for sentence-level semantics, and employs an attention-based point cloud encoder for spatial understanding. We further propose Masked Instance Training (MIT) to filter out non-aligned objects and improve multimodal robustness. To enhance the embedding space, we introduce Modality-aware Hierarchical Contrastive Learning (MHCL), incorporating cross-modal, submap-, text-, and instance-level losses. In the fine localization stage, we completely remove explicit text-instance matching and design a lightweight yet powerful framework based on Prototype-based Map Cloning (PMC) and a Cascaded Cross-Attention Transformer (CCAT). Extensive experiments on the KITTI360Pose dataset show that Text2Loc++ outperforms existing methods by up to 15%. In addition, the proposed model exhibits robust generalization when evaluated on the new dataset, effectively handling complex linguistic expressions and a wide variety of urban environments. The code and dataset will be made publicly available.