Unconstrained Matching of 2D and 3D Descriptors for 6-DOF Pose Estimation

TL;DR

This paper introduces a novel method for directly matching 2D image features with 3D point cloud features to estimate camera pose, demonstrating high accuracy in diverse indoor and outdoor scenarios.

Contribution

It presents a new approach to directly match 2D and 3D descriptors for 6-DOF pose estimation, bypassing traditional registration steps.

Findings

High-precision pose estimation across various environments

Effective matching of 2D and 3D descriptors in unconstrained settings

Robustness demonstrated on different types of point clouds

Abstract

This paper proposes a novel concept to directly match feature descriptors extracted from 2D images with feature descriptors extracted from 3D point clouds. We use this concept to directly localize images in a 3D point cloud. We generate a dataset of matching 2D and 3D points and their corresponding feature descriptors, which is used to learn a Descriptor-Matcher classifier. To localize the pose of an image at test time, we extract keypoints and feature descriptors from the query image. The trained Descriptor-Matcher is then used to match the features from the image and the point cloud. The locations of the matched features are used in a robust pose estimation algorithm to predict the location and orientation of the query image. We carried out an extensive evaluation of the proposed method for indoor and outdoor scenarios and with different types of point clouds to verify the feasibility of our approach. Experimental results demonstrate that direct matching of feature descriptors from images and point clouds is not only a viable idea but can also be reliably used to estimate the 6-DOF poses of query cameras in any type of 3D point cloud in an unconstrained manner with high precision.