2-Entity RANSAC for robust visual localization in changing environment

TL;DR

This paper introduces a robust 2-entity RANSAC method for visual localization that effectively handles environmental changes and high outlier ratios by utilizing combined line and point features with minimal solutions.

Contribution

The paper proposes a novel 2-entity RANSAC approach with minimal closed-form solutions for 3D-2D localization using inertial data, improving robustness to outliers and environmental variations.

Findings

Enhanced robustness to outliers in changing environments

Effective use of combined line and point features

Validated performance on synthetic and real datasets

Abstract

Visual localization has attracted considerable attention due to its low-cost and stable sensor, which is desired in many applications, such as autonomous driving, inspection robots and unmanned aerial vehicles. However, current visual localization methods still struggle with environmental changes across weathers and seasons, as there is significant appearance variation between the map and the query image. The crucial challenge in this situation is that the percentage of outliers, i.e. incorrect feature matches, is high. In this paper, we derive minimal closed form solutions for 3D-2D localization with the aid of inertial measurements, using only 2 pairs of point matches or 1 pair of point match and 1 pair of line match. These solutions are further utilized in the proposed 2-entity RANSAC, which is more robust to outliers as both line and point features can be used simultaneously and the number of matches required for pose calculation is reduced. Furthermore, we introduce three feature sampling strategies with different advantages, enabling an automatic selection mechanism. With the mechanism, our 2-entity RANSAC can be adaptive to the environments with different distribution of feature types in different segments. Finally, we evaluate the method on both synthetic and real-world datasets, validating its performance and effectiveness in inter-session scenarios.

Figures

Tables

Table 1. TABLE I: Evaluation of the localization on the selected scene of YQ-Dataset.

	2017-0827		2017-0828		2018-0129
	unstructured scene	structured scene	unstructured scene	structured scene	unstructured scene	structured scene
Translation Error (m) Rotation Error (deg)	.25/.5/1.0/5.0 2/5/8/10	.25/.5/1.0/5.0 2/5/8/10	.25/.5/1.0/5.0 2/5/8/10	.25/.5/1.0/5.0 2/5/8/10	.25/.5/1.0/5.0 2/5/8/10	.25/.5/1.0/5.0 2/5/8/10
1P1L-4DoF	0.0 / 01.9 / 03.8 / 15.0	07.9 / 23.7 / 42.1 / 81.6	01.9 / 09.4 / 15.0 / 29.4	13.0 / 32.6 / 54.3 / 87.0	0.0 / 0.0 / 0.0 / 3.8	05.0 / 15.5 / 35.5 / 85.0
1P1L-6DoF	1.9 / 05.7 / 08.8 / 18.8	29.5 / 48.6 / 78.4 / 91.9	06.3 / 13.1 / 21.9 / 35.0	39.1 / 67.4 / 82.6 / 91.3	0.0 / 0.0 / 0.0 / 4.9	25.0 / 45.0 / 65.0 / 90.0
2P-4DoF	2.5 / 11.3 / 18.8 / 36.3	20.5 / 31.8 / 50.0 / 70.5	11.3 / 29.4 / 42.5 / 56.3	21.7 / 52.2 / 67.4 / 93.5	0.0 / 0.0 / 1.0 / 6.8	10.0 / 20.0 / 30.0 / 80.0
2P-6DoF	6.3 / 15.0 / 24.4 / 40.0	31.8 / 57.9 / 83.9 / 86.8	25.0 / 38.1 / 48.8 / 58.1	45.7 / 69.6 / 91.3 / 95.7	1.0 / 1.0 / 1.0 / 9.7	25.0 / 40.0 / 65.0 / 95.0
mixed-4DoF	4.4 / 13.8 / 26.9 / 44.4	13.6 / 31.8 / 47.7 / 72.7	10.6 / 30.0 / 37.5 / 56.9	21.7 / 43.5 / 63.0 / 95.7	0.0 / 0.0 / 1.0 / 7.8	05.0 / 25.0 / 35.0 / 80.0
mixed-6DoF	6.9 / 15.0 / 23.4 / 38.8	31.8 / 63.2 / 78.9 / 89.6	25.6 / 36.9 / 48.1 / 58.1	45.7 / 73.9 / 93.5 / 97.8	1.0 / 1.0 / 1.0 / 7.8	20.0 / 55.0 / 70.0 / 95.0
P3P	1.9 / 11.9 / 20.0 / 37.5	29.5 / 50.0 / 73.7 / 86.8	16.3 / 26.9 / 39.4 / 52.5	30.4 / 63.0 / 76.1 / 91.3	0.0 / 0.0 / 1.0 / 2.9	10.0 / 40.0 / 60.0 / 90.0
EPnP	4.4 / 11.9 / 20.6 / 35.7	27.3 / 55.3 / 81.6 / 89.5	16.9 / 31.3 / 40.6 / 56.3	26.1 / 65.2 / 84.8 / 93.5	0.0 / 0.0 / 1.0 / 2.9	20.0 / 40.0 / 65.0 / 90.0

Table 2. TABLE II: Evaluation on the whole YQ-Dataset.

	2017-0827	2017-0828	2018-0129
Trans Error (m) Rota Error (deg)	.25/.5/1.0/5.0 2/5/8/10	.25/.5/1.0/5.0 2/5/8/10	.25/.5/1.0/5.0 2/5/8/10
1P1L-4DoF	02.1 / 05.2 / 09.7 / 21.5	02.6 / 07.6 / 14.3 / 26.1	2.5 / 06.0 / 10.6 / 21.8
1P1L-6DoF	06.3 / 10.2 / 15.3 / 24.3	07.7 / 14.4 / 20.4 / 28.6	6.0 / 09.3 / 12.5 / 23.7
2P-4DoF	03.6 / 10.0 / 17.2 / 31.1	08.6 / 19.3 / 28.0 / 39.3	0.4 / 05.9 / 12.6 / 25.4
2P-6DoF	09.3 / 17.1 / 24.3 / 34.4	15.7 / 25.3 / 32.5 / 39.8	5.9 / 11.5 / 16.6 / 27.3
mixed-4DoF	05.7 / 14.4 / 22.4 / 34.1	07.1 / 17.3 / 25.6 / 35.5	3.9 / 09.3 / 15.6 / 28.5
mixed-6DoF	12.1 / 18.7 / 24.4 / 33.3	13.7 / 21.9 / 28.5 / 35.1	7.3 / 12.6 / 17.0 / 27.7
P3P	06.7 / 14.7 / 21.3 / 32.2	10.1 / 19.8 / 28.2 / 37.1	4.3 / 09.7 / 12.9 / 22.8
EPnP	07.6 / 16.6 / 23.5 / 32.3	09.6 / 20.3 / 28.4 / 38.4	5.3 / 11.1 / 15.0 / 23.8

Table 3. TABLE III: The automatic model selection experiment.

Method	1P1L 4DoF	1P1L 6DoF	2P 4DoF	2P 6DoF	mixed 4DoF	mixed 6DoF	selection
AUC	0.51	0.58	0.78	0.81	0.71	0.72	0.84

Equations

$$T_{{\mathcal{W}}_m{\mathcal{C}}_q}=[R_z(\alpha )R_y(\tilde{\beta })R_x(\tilde{\gamma })|(\begin{array}{ccc}{T}_1& T_2& T_3\end{array}{)}^T]$$

$$C=\left[\begin{array}{c}0\\ 0\\ -1\end{array}\right],D_2=\left[\begin{array}{c}0\\ 0\\ 0\end{array}\right],D_3=\left[\begin{array}{c}\tan (\arccos (\overrightarrow{d_2}\cdot \overrightarrow{d_3}))\\ 0\\ 0\end{array}\right]$$

$$C^0=0_{3\times 1},D_2^0=C^0+\overrightarrow{d_2},D_3^0=C^0+\frac{\overrightarrow{d_3}}{\overrightarrow{d_2}\cdot \overrightarrow{d_3}}$$

$$T_{{\mathcal{W}}_1{\mathcal{W}}_m}=\left[\begin{array}{cc}{\mathbf{I}}_{3\times 3}& -P_1^0\\ 0_{1\times 3}& 1\end{array}\right]$$

$$P_1=0_{3\times 1},L_{i=\{2,3\}}\triangleq {\left[\begin{array}{ccc}{X}_i& Y_i& Z_i\end{array}\right]}^T$$

$$a_1{T}_2-b_1{T}_1=0$$

$$b_1{T}_3-T_2=-b_1$$

$$R_{21}{X}_2+R_{22}{Y}_2+R_{23}{Z}_2+T_2=0$$

$$R_{21}{X}_3+R_{22}{Y}_3+R_{23}{Z}_3+T_2=0$$

$$T_{{\mathcal{C}}_q{\mathcal{W}}_m}={T_{{\mathcal{C}}_1{\mathcal{C}}_q}}^{-1}\cdot T_{{\mathcal{C}}_1{\mathcal{W}}_1}\cdot T_{{\mathcal{W}}_1{\mathcal{W}}_m}$$

$$C=0_{3\times 1},D_{i=\{1,2\}}\triangleq {\left[\begin{array}{ccc}{a}_i& b_i& 1\end{array}\right]}^T$$

$$P_1=0_{3\times 1},P_2\triangleq {\left[\begin{array}{ccc}{X}_2& Y_2& Z_2\end{array}\right]}^T$$

$$a_1{T}_2-b_1{T}_1=0$$

$$a_1{T}_3-T_1=0$$

$$\begin{array}{rl}& a_2(R_{21}{X}_2+R_{22}{Y}_2+R_{23}{Z}_2+T_2)\\ & -b_2(R_{11}{X}_2+R_{12}{Y}_2+R_{13}{Z}_2+T_1)=0\end{array}$$

$$\begin{array}{rl}& a_2(R_{31}{X}_2+R_{32}{Y}_2+R_{33}{Z}_2+T_3)\\ & -(R_{11}{X}_2+R_{12}{Y}_2+R_{13}{Z}_2+T_1)=0\end{array}$$

$$T_{{\mathcal{C}}_q{\mathcal{W}}_m}=T_{{\mathcal{C}}_1{\mathcal{W}}_1}\cdot T_{{\mathcal{W}}_1{\mathcal{W}}_m}$$

$$\overrightarrow{n_1}=\overrightarrow{d_1}\times \overrightarrow{d_2},\overrightarrow{n_2}=\overrightarrow{d_3}\times \overrightarrow{d_4},\overrightarrow{d_{12}}=\overrightarrow{n_1}\times \overrightarrow{n_2}$$

$$D_1={\left[\begin{array}{ccc}\tan (\arccos (\overrightarrow{d_1}\cdot \overrightarrow{d_{12}}))& 0& 0\end{array}\right]}^T$$

$$C^0=0_{3\times 1},D_1^0=C^0+\frac{\overrightarrow{d_1}}{\overrightarrow{d_1}\cdot \overrightarrow{d_{12}}},D_2^0=C^0+\overrightarrow{d_{12}}$$

$$L_1=0_{3\times 1},L_{\{i=2,3,4\}}\triangleq {\left[\begin{array}{ccc}{X}_i& Y_i& Z_i\end{array}\right]}^T$$

$$T_2=0$$

$$R_{21}{X}_2+R_{22}{Y}_2+R_{23}{Z}_2+T_2=0$$

$$\begin{array}{rl}& a_1(R_{21}{X}_3+R_{22}{Y}_3+R_{23}{Z}_3+T_2)\\ & -b_1(R_{11}{X}_3+R_{12}{Y}_3+R_{13}{Z}_3+T_1)=0\end{array}$$

$$\begin{array}{rl}& a_1(R_{21}{X}_4+R_{22}{Y}_4+R_{23}{Z}_4+T_2)\\ & -b_1(R_{11}{X}_4+R_{12}{Y}_4+R_{13}{Z}_4+T_1)=0\end{array}$$

$$\frac{m}{p}=\lambda ,\frac{n}{l}=\gamma (0\le \lambda ,\gamma \le 1),\frac{l}{p}=\epsilon (0\le \epsilon \le l)$$

$$P_{1p1l}$$

$$P_{2p}$$

$$P_{mixed}$$

$$P_{1p1l}-P_{mixed}=\lambda (\gamma -\frac{\lambda p+\gamma l-1}{p+l-1})\propto \gamma -(\lambda -a)$$

$$P_{mixed}-P_{2p}=\lambda (\frac{\lambda p+\gamma l-1}{p+l-1}-\frac{\lambda p-1}{p-1})\propto \gamma -(\lambda -a)$$

$$\gamma \ge \lambda \Rightarrow P_{1p1l}>P_{mixed}>P_{2p}$$

Full text

I Introduction

Localization is essential for autonomous robot navigation. Compared to the LiDAR-based localization methods, visual localization is favorable in many applications as the low-cost and stable cameras are widely available. A typical workflow of the visual localization task is to detect the feature points in the query image (FAST [1], SIFT [2]), match the image feature points to the map points (ORB [3], SIFT [2]), and estimate the pose based on a set of matches. Match outliers are hardly avoidable in this setting, leading to inaccurate pose estimation. Random sample consensus (RANSAC) [4] is a popular method to achieve robust estimation by randomly sampling the matching set and voting for the inliers. However, RANSAC is limited by serious appearance changes in the environment, in which the percentage of outliers may grow significantly [5][6]. Therefore, reliable visual localization robust to the weather, illumination or seasonal changes remains a challenging problem.

Two factors are crucial for RANSAC to find inliers, i.e. the number of inliers in the set (higher is better), and the minimal number of matches required to estimate a pose (lower is better). For the first factor, some previous works propose to increase the number of inliers by utilizing multiple types of features (e.g. points, lines and planes)[7][8], as the feature defined on larger image regions usually leads to higher robustness against the illuminative variation. In this paper, we also consider multiple types of features following the practice of previous methods [9]. For the second factor, the well-known 3D-2D localization methods typically require 3 or 4 feature matches such as perspective-3-points (P3P) [10], efficient perspective-n-points (EPnP) [11], the minimal solutions of 3 lines [12][13] or a mixture of 3 entities (i.e. points and lines) [9]. Our key observation of this paper is that the number of matches can be further reduced for both point and line features in robotics navigation scenario, leading to higher robustness against outliers.

Specifically, with the growing trend of visual inertial navigation in robots, the direction of gravity between the pre-built map and the current query image can be easily aligned. Therefore, we aim to reduce the number of point feature matches from 3 to 2 for closed form pose estimation, by making use of the directly observed pitch and roll angle. Furthermore, the closed form solution is also derived for the case of 1 point and 1 line, so that line feature matches can be considered. As a result, we propose a 2-entity RANSAC for persistent visual localization as illustrated in Fig. 1. The 2-entity is named for that only two feature matches between points and lines are used to pose estimation. In summary, we present the contributions of the paper as follows:

• •

We propose a general framework to derive the closed form solutions to pose estimation which can deal with both point and line features, namely 2 point matches or 1 point match and 1 line match.

• •

We propose a 2-entity RANSAC by embedding the minimal closed form solutions into the RANSAC framework and proposed three sampling strategies in RANSAC that select different types of feature matches for pose estimation.

• •

We analyze the success probability for different sampling strategies in the 2-entity RANSAC, and propose a selection mechanism that adaptively select strategy depending on environmental characteristics, i.e. structured or unstructured.

• •

The proposed method is evaluated on both synthetic and multiple real world data sessions. The results validate the effectiveness and efficiency of our method on visual localization with seasonal and illuminative changes.

The remainder of this paper are organized as follows: In Section II we review some relevant visual pose estimation methods. Section III gives the overview of the proposed method and the derivation of the closed form solutions. The analysis about the criterion to select the sampling strategy in RANSAC is addressed in Section IV. We then evaluate our method in Section V on synthetic data and challenging real world data. Finally, the conclusion is drawn in Section VI, which completes this paper.

II Related Works

The minimal solution with efficient 2D-2D point matches for camera pose estimation has been studied extensively for decades. There are many well-known methods, such as 8-point method for fundamental matrix estimation which yields a unique solution [14]. In addition, there are also 7-point [15], 6-point [16] and 5-point [17] methods which focus on further reduction of minimal number of point matches required for pose estimation. In recent years, with the development of visual inertial navigation, this number is further reduced by considering the measurement of angular velocity and the gravity. Based on this, a 3-point method was proposed in [18]. More recently, the 2-point RANSAC [19][20] was proposed by employing the gyro to calculate the relative rotation within short duration, which was employed in several visual inertial odometry softwares. By further constraining the motion pattern of the camera, only 1 point match is sufficient to derive the pose [21].

In localization, 3D-2D matches are available but there is no measurement of relative yaw angle. In this case, the minimal number of required feature matches is reduced compared with scenario of 2D-2D. Specifically, there are several 3-point methods [10][22][23][24][25] that give up to four solutions in closed-form and other 3-line solvers [12][13] that give up to eight solutions which require slower iterative methods. As significant change can appear in long term, it’s difficult to find enough point matches or line matches alone to guarantee the success rate of localization. Thus the methods utilizing both point and line features are studied in [9] including two possible cases: 2 points 1 line case and 1 point 2 lines case. With the aid of inertial measurement,the 2-point RANSAC solution is investigated in [20] and [26]. While the former utilized the relative measurement of yaw angle, which is only available between two consecutive camera under odometry problem, thus cannot be applied in localization, and the latter only considers point feature matches. To the best of our knowledge, the minimal solution for localization using both point and line features with the aid of inertial measurements has not been studied yet, which is the focus of this paper.

III Minimal Solutions

The definition of coordinates in this paper is shown in Fig. 1. A 3D map is built by running a visual inertial simultaneous localization and mapping (VI-SLAM), of which the world reference frame is denoted as ${\mathcal{W}_{m}}$. In the current query session, a visual inertial navigation system (VINS) is utilized with its origin defined as ${\mathcal{W}_{q}}$. Denoting the coordinates of the current query camera as ${\mathcal{C}_{q}}$, the localization problem is stated as the estimation of the query image pose with respect to the world, i.e. $T_{{\mathcal{W}_{m}}{\mathcal{C}_{q}}}$. Thanks to the observability of pitch and roll angle, $\tilde{\beta}$ and $\tilde{\gamma}$, given by the inertial measurements, the direction of gravity can be aligned between ${\mathcal{W}_{m}}$ and ${\mathcal{W}_{q}}$ [27], leading to the reduction of degrees of freedom (DoF) in pose as

[TABLE]

where $\alpha$ and $(T_{1},T_{2},T_{3})$ denote the yaw angle and three translations to be estimated. In sequel, for the purpose of clearance, we are going to estimate the inversion of the pose, i.e. ${T_{{\mathcal{C}_{q}}{\mathcal{W}_{m}}}}$. Note that the number of unknown DoFs in ${T_{{\mathcal{C}_{q}}{\mathcal{W}_{m}}}}$ remains the same.

To solve the 4DoF localization problem, we refer to two geometric constraints: collinearity of 3D-2D point matches and coplanarity of 3D-2D line matches. In an image, each pixel encodes a 3D projection ray. As shown in Fig. 2, according to the projection geometry, the optical center $C^{0}$ of the camera, the map point $P_{1}^{0}$ and its projection point $D_{1}^{0}$ on the image lie on the same 3D line, which is denoted as $\{C^{0},D_{1}^{0},{R_{{\mathcal{C}_{q}}{\mathcal{W}_{m}}}}P_{1}^{0}+{t_{{\mathcal{C}_{q}}{\mathcal{W}_{m}}}}\}_{L}$. By solving the line equation with the first two points $C^{0}$ and $D_{1}^{0}$, the third point can be substituted into the line equation to form two independent constraints. In addition, the camera center $C^{0}$, the map line $L_{2}^{0}L_{3}^{0}$ and its projection line segment $D_{2}^{0}D_{3}^{0}$ lie on the single plane $\pi$, which is denoted as ${\{C^{0},{D_{2}^{0}},{D_{3}^{0}},{R_{{\mathcal{C}_{q}}{\mathcal{W}_{m}}}}{L_{2}^{0}}+{t_{{\mathcal{C}_{q}}{\mathcal{W}_{m}}}}\}_{P}}$, and ${\{C,{D_{2}^{0}},{D_{3}^{0}},{R_{{\mathcal{C}_{q}}{\mathcal{W}_{m}}}}{L_{3}^{0}}+{t_{{\mathcal{C}_{q}}{\mathcal{W}_{m}}}}\}_{P}}$. Similarly, by solving the plane equation from the first three points $C^{0}$, $D_{2}^{0}$ and $D_{3}^{0}$, the two end points of the map line can be substituted into it to form another two independent constraints. As a result, two non-degenerate feature matches should be sufficient to solve the 4DoF localization problem with three possible combinations: 1 point 1 line, 2 points and 2 lines in theory.

III-A 1 point 1 line

In this section, we are going to solve the 4DoF localization problem given one 3D-2D point match and one 3D-2D line match. The solution is denoted as 1P1L. Given the 2D coordinates of the detected image feature point, we can compute the corresponding 3D projection ray according to the calibrated intrinsic parameters. The same process is applied to the two end points of the line segment to obtain two 3D projection rays. The 3D projection rays are expressed in the camera reference frame $\mathcal{C}_{q}$, and the corresponding 3D coordinates of the features are expressed in the world reference frame $\mathcal{W}_{m}$. To simplify the form of the unsolved equations, we introduce two intermediate reference frames denoted as $\mathcal{C}_{1}$ and $\mathcal{W}_{1}$ for camera and map respectively.

The choice of $\mathcal{C}_{1}$: As shown in Fig. 2 (a), in the original camera reference frame $\mathcal{C}_{q}$, the origin is the camera center $C^{0}$, the camera projection ray associated with the 2D point is given by its normalized direction vector $\overrightarrow{d_{1}}$, and projection rays of the two end points of the 2D line are given by normalized direction $\overrightarrow{d_{2}}$, $\overrightarrow{d_{3}}$.

In the intermediate camera frame $\mathcal{C}_{1}$, the projection ray associated with the 2D point is denoted by $CD_{1}$, and the line, $CD_{2}$ and $CD_{3}$. Specifically, the intermediate reference frame $\mathcal{C}_{1}$ should satisfy the following conditions:

• •

The new camera center $C$ is $(0,0,-1)$.

• •

One of the projection rays of the line end point $CD_{2}$ lies on the $z$ axis such that $D_{2}=(0,0,0)$.

• •

The other projection ray of the line end point $CD_{3}$ lies on the $xz$ plane and the point $D_{3}$ is the intersection point between the $x$ axis and the ray.

• •

The point $D_{1}$ in the projection ray of the point feature lies on the $xy$ plane.

After the transformation, we have following results

[TABLE]

Then the points in the original camera frame are calculated as follows

[TABLE]

The transformation $T_{{\mathcal{C}_{1}}{\mathcal{C}_{q}}}$ can be computed by transforming the three points $(C^{0},D^{0}_{2},D^{0}_{3})$ to $(C,D_{2},D_{3})$. After that, the point $D_{1}\triangleq(a_{1},b_{1},0)$ can also be computed.

The choice of $\mathcal{W}_{1}$: The transformation of the world reference is a translation which transforms the 3D point to the origin of $\mathcal{W}_{1}$

[TABLE]

Thus in $\mathcal{W}_{1}$ we have:

[TABLE]

Note that these points are all known.

Pose estimation between $\mathcal{C}_{1}$ and $\mathcal{W}_{1}$: Let us denote the rotation matrix and the translation matrix to be solved as $R$ and $t$, that is $R\triangleq{R_{{\mathcal{C}_{1}}{{\mathcal{W}}_{1}}}}$, $t\triangleq{t_{{\mathcal{C}_{1}}{{\mathcal{W}}_{1}}}}$. According to the collinearity of ${\{C,D_{1},RP_{1}+t\}}_{L}$, the following equations are derived:

[TABLE]

[TABLE]

As for the coplanarity of ${\{C,D_{2},D_{3},RL_{2}+t\}}_{P}$:

[TABLE]

And the coplanarity of ${\{C,D_{2},D_{3},RL_{3}+t\}}_{P}$:

[TABLE]

where $R_{mn}$ denotes the $m$-th row and $n$-th column entry of $R$, $T_{m}$ denotes the $m$-th entry of $t$.

One should notice that the matrix $R$ is determined only by $\alpha$. Combining (3) and (4), we can solve $\alpha$, which is then substituted in (1) - (4) for the translation $t$. Put all the equations together, the closed form localization problem can be solved as

[TABLE]

Degenerate cases: For 1 point 1 line case, if the point lies on the line, the corresponding case is degenerated.

III-B 2 points

In this section, we are given two 3D-2D point matches, and the solution is denoted as 2P. In this case, the intermediate camera reference frame remains the same as the original one, thus the points in $\mathcal{C}_{q}$ and $\mathcal{C}_{1}$ following the notations in 1P1L are

[TABLE]

where $a_{1},a_{2},b_{1},b_{2}$ are known parameters computed using intrinsic parameters and normalized depth.

The transformation of the world reference is a translation which transforms one of the two 3D points to the origin of the intermediate world frame. Thus in $\mathcal{W}_{1}$ we have:

[TABLE]

Pose estimation between $\mathcal{C}_{1}$ and $\mathcal{W}_{1}$: Following the notations in 1P1L, according to the collinearity of $\{C,D_{1},RP_{1}+t\}_{L}$, the following equations are derived:

[TABLE]

[TABLE]

As for the collinearity of $\{C,D_{2},RP_{2}+t\}_{L}$, we have:

[TABLE]

[TABLE]

Combining (5) - (8), we can solve ${T_{{\mathcal{C}_{1}}{{\mathcal{W}}_{1}}}}$. Thus the localization result can be obtained by

[TABLE]

III-C 2 lines

In this section, we are given two 3D-2D line matches as shown in Fig. 2 (b).

The choice of $\mathcal{C}_{1}$: In $\mathcal{C}_{q}$, the camera projection rays associated with the two 2D lines are given by pairs $(\overrightarrow{d_{1}},\overrightarrow{d_{2}})$ and $(\overrightarrow{d_{3}},\overrightarrow{d_{4}})$, respectively. In $\mathcal{C}_{1}$, the 3D projection rays associated with the two 2D lines are represented by $(CD_{1},CD_{2})$, and $(CD_{2},CD_{3})$. Specifically, $\mathcal{C}_{1}$ should satisfy the following conditions:

• •

The new camera center $C$ is $(0,0,-1)$.

• •

The intersection line of the two interpretation planes represented by projection ray $CD_{2}$ lies on the $z$ axis such that $D_{2}=(0,0,0)$.

• •

The projection ray $CD_{1}$ lies on the $xz$ plane and the point $D_{1}$ is the intersection point between the $x$ axis and the ray.

• •

The point $D_{3}$ in the projection ray of one line lies on the $xy$ plane.

The unit normal vectors of the two planes formed by $(C^{0},\overrightarrow{d_{1}},\overrightarrow{d_{2}})$ and $(C^{0},\overrightarrow{d_{3}},\overrightarrow{d_{4}})$ can be computed as follows

[TABLE]

where $\overrightarrow{d_{12}}$ is the direction vector of the intersection line $CD_{2}$. After such a transformation, we have the coordinates of $C$ and $D_{2}$, and $D_{1}$ can be computed as follows

[TABLE]

Then the corresponding points in $\mathcal{C}_{q}$:

[TABLE]

The transformation $T_{{\mathcal{C}_{1}}{\mathcal{C}_{q}}}$ can be computed by transforming $(C^{0},D^{0}_{1},D^{0}_{2})$ to $(C,D_{1},D_{2})$. After that, the point $D_{3}\triangleq(a_{1},b_{1},0)$ can also be computed.

The choice of $\mathcal{W}_{1}$: $T_{{\mathcal{W}_{1}}{\mathcal{W}_{m}}}$ is a translation which transforms one end point of the 3D line to the origin of the intermediate world frame. Thus in $\mathcal{W}_{1}$ we have

[TABLE]

Pose estimation between $\mathcal{C}_{1}$ and $\mathcal{W}_{1}$: Following the notations in 1P1L, according to the coplanarity of $\{C,D_{1},D_{2},RL_{1}+t\}_{P}$ and $\{C,D_{1},D_{2},RL_{2}+t\}_{P}$, the following equations are derived

[TABLE]

[TABLE]

As for the coplanarity of ${\{C,D_{2},D_{3},RL_{3}+t\}}_{P}$ and ${\{C,D_{2},D_{3},RL_{4}+t\}}_{P}$:

[TABLE]

[TABLE]

From (9) - (12), we can easily find that the constraints are not sufficient to solve $T_{3}$. In fact, one of the constraints provided by IMU is coincident with one of constraints provided by coplanarity, thus the 2 lines case cannot solve the 4DoF localization problem.

IV Model Selection

In the last section, we discussed the possible minimal closed form solutions of 4DoF localization problem when inertial measurements exist, and showed that the 1P1L and 2P are solvable. Based on the two minimal solutions, we propose the 2-entity RANSAC, in which there are three possible sampling strategies according to the corresponding solution: 1P1L, 2P and mixed. The mixed refers to using both 1P1L and 2P solutions according to the randomly selected features. Specifically, we first select one point match and then randomly select another feature match from points and lines. If it is a point match, we perform 2P, if it is a line match, we perform 1P1L. There are some works discuss about sampling points from multiple combinations of views [28] [29], while we are focusing on sampling from multiple types of features (points and lines). In this section, we derive the success probability of the three sampling strategies to provide insights of sampling in 2-entity RANSAC.

Let’s denote $p$ as the number of point matches, $l$ as the number of line matches, $\lambda$ as the point inliers rate, and $\gamma$ as the line inliers rate.

[TABLE]

where $m$, $n$ denote the point inliers number and line inliers number respectively. Then the success probability of different sampling strategies during one iteration in RANSAC is derived as follows:

[TABLE]

Then we have

[TABLE]

where $a=\frac{1-\lambda}{p-1}$.

Generally, $p-1\gg 1-\lambda$ holds for most real world applications, which means $a$ is a small positive number close to 0. Thus, the following conclusion can be derived:

[TABLE]

Furthermore, with proper scaling, the following conclusion can also hold:

[TABLE]

The percentage of line features and point features is highly related to the environment. According to the above discussion, we could find that the mixed sampling strategy is a more robust selection to the environmental variations as its performance is always the average one. Empirically, when localizing in the unstructured scene, there might be more point feature matches which indicates a higher point inlier rate than line features, leading to the choice of 2P sampling strategy. In the structured scene, although there are more line features compared with the unstructured scene, there are also many good point features. Considering the reconstruction error of the 3D lines can be higher than that of 3D points, the performance of 1P1L may not be significantly better than the 2P method in the structured scene. In order to be adaptive to the changing environment, an automatic selection mechanism is proposed by statistic estimation of the inlier rate using historical data. Specifically, the map is divided into structured or unstructured segments according to the inlier rate of point and line features. Then, when the robot is in the segment, the corresponding sampling strategy is selected, e.g. 2P in unstructured environment.

V Experimental Results

The proposed methods are evaluated on both synthetic and real data to demonstrate the advantages over other baseline methods in visual localization. We start by simulation experiments to illustrate the accuracy when the image feature is noisy, the sensitivity with respect to inaccurate pitch and roll angle, and the robustness against outliers. Then on real world data, we demonstrate the success rate comparison to show the effectiveness of the proposed methods and the model selection mechanism.

V-A Results with synthetic data

In the simulation experiments, we generated some 3D points and lines in the cube $[-1,1]^{3}$ and computed 2D projections for varying camera poses to get 3D-2D feature matches. For each method, 100 iterations of RANSAC are performed. The final identified inliers are sent to a nonlinear minimization for high accuracy. In optimization, there are two possible ways: 4DoF or 6DoF optimization. The 4DoF optimization means we fix the pitch and roll angle provided by IMU, and only optimize the other four variables in the pose. This may be useful when there are little inliers to optimize.

The minimal solutions evaluated in simulation experiments are EPnP[11], P3P[10], 2P1L[9] and our proposed solutions including 1P1L-6DoF, 1P1L-4DoF, 2P-6DoF, 2P-4DoF. The mixed-6DoF and mixed-4DoF are also evaluated in the robustness against outliers experiment. We exploited the translation and rotation error to evaluate the estimated result $[R|t]$ with respect to the ground truth $[R_{gt}|t_{gt}]$. Following [9], for translation error, we computed $\|t-t_{gt}\|/t_{gt}$ in percentage. For rotation error, we first computed the relative rotation $\triangle R=RR_{gt}^{T}$ and represented it in degrees.

V-A1 Accuracy

To quantify the accuracy of different minimal solutions, we added Gaussian noise with zero mean and varying standard deviations for the 2D projections and changed the number of feature matches in four levels: 10, 5, 4, 3 (the 10 case means there are 10 point matches and 10 line matches in the scene). The results are shown in Fig. 3. We could find that when the feature matches are sufficient (see the 10 case), the proposed methods can achieve the same accuracy over other baseline methods as the standard deviation of the noise increases. While the number of feature matches decreases, the advantage of our 2-entity methods becomes bigger compared with 2P1L, EPnP and P3P. One can notice that when the number of feature matches decreases to 4, the error of our methods is slightly larger than that of the 10 matches case, while others grow severely.

V-A2 Sensitivity

With the inertial sensor’s aid, we reduce the degree of the localization problem to 4 by leveraging the pitch and roll angle provided by the sensor. Therefore, it’s necessary to investigate the influence of the quality of the two angles on the final accuracy. We added Gaussian noise with zero mean and varying standard deviations on the pitch and roll angle and studied the performance of the proposed methods in three feature matches levels: 10, 5, 3. The results are shown in Fig. 4.

When there are enough feature matches (see the 10 case), the proposed methods can tolerate almost 25 degrees of noise on both pitch and roll angles. Empirically in real world application, the noise on pitch and roll angle is far less, thus the noise on pitch and roll should have limited impact on the accuracy. We also note that the 6DoF optimization outperforms the 4DoF in this case. However, as the number of feature matches decreases, the difference between the 6DoF and 4DoF methods becomes smaller. This is reasonable because when the inliers number is few, the error caused by the additional DoFs in optimization may be larger than that caused by noisy attitude estimation. Therefore, if there are very limited reliable feature matches when long term change occurs, 4DoF optimization is a good choice.

V-A3 Robustness

To validate the robustness of the proposed methods, we designed a few experiments by varying the outlier rate from 0 to 80% with different number levels of inliers: 10, 5, 4, 3. We achieve the outlier rate by adding outliers and the outlier is generated by incorrectly associating the features in the original data. When the translation error is less than 10% and the rotation error is smaller than 5 degrees, the localization is assumed to be successful. We did 200 trails for each method to average the success rate. The results can be seen in Fig. 5. As expected, the proposed 2-entity RANSAC outperforms the other minimal solutions when the outlier rate increases, which is more obvious when the number of inliers decreases. Note that with sufficient inliers, the proposed methods can achieve a success rate of more than 90% when the outlier rate is 80%.

V-B Results with real data

For real world experiments, YQ-Dataset is utilized, and three of the datasets were collected during three days in summer 2017, denoted as 2017-0823, 2017-0827, 2017-0828, and the other was collected in winter 2018, denoted as 2018-0129. We select the 2017-0823 dataset to build the 3D map and utilize the other three datasets to evaluate the localization performance over different weathers or seasons. The ground truth relative pose is provided by aligning the synchronized 3D LiDAR scans. The evaluated methods are EPnP, P3P, and our methods including 1P1L-4/6DoF, 2P-4/6DoF, mixed-4/6DoF. To get the 3D-2D feature matches between the query image and the map, we exploited the following steps:

• •

Obtain the camera poses and the 3D-2D point matches in the map using visual inertial SLAM software [30].

• •

Run Line3D++ algorithm [31] to get the 3D-2D line matches in the map.

• •

For the query session, we get the 3D-2D points/lines match based on the descriptors of LibVISO2 [32] and LBD [33], which are fed to the RANSAC with different solutions.

• •

For the pitch and roll angle of the query image, we directly use the corresponding values measured by IMU.

We first counted the correctly localized query images out of all keyframes in the query session using different methods, and represent the successful localization rate in percentage under four translation and rotation error thresholds, which can be seen in Tab. II. As expected, the performance of methods with 6DoF optimization are still better than 4DoF optimization. Besides, the 2-entity RANSAC is better than RANSAC with P3P and EPnP which calls for more number of matches. The proposed 2P and mixed are obviously better than EPnP and P3P, which indicates the robustness of the proposed methods when dealing with different weathers and seasons. Note that 1P1L is not as good as point features based methods, of which the reason is that there are many trees on both sides of the road in the map environment so that point features are far more abundant than line features.

In order to make a fair comparison between the 1P1L and the 2P methods, we manually picked some structured and unstructured segments from the map, in which some images are shown in Fig. 6. There are around 100 places selected in unstructured segments, and 50 places in structured segments. The success rates of localization with multiple methods on the selected segments are show in Tab. I. The results show that the 2P method performs best in the unstructured scene and the performance of 1P1L is similar with 2P in the structured scene but obviously worse than that of 2P in unstructured segments, confirming the fact that the feature distribution in the environment has unavoidable effect on the method selection. Besides, from the results of the structured scene in 2018-0129, we can find that when outlier percentage grows due to the changing season, the better robustness of line feature matches promotes the performance of 1P1L method. The mixed-6DoF, as derived in theory, gives relatively stable performance compared with 2P and 1P1L in all segments.

To further verify the correctness of the model selection mechanism, we did an extra leave-one-out experiment on the 2017-0828 dataset. According to the localization results of 2017-0827 and 2018-0129 dataset on the 2017-0823 map, we automatically labeled the whole dataset with structured and unstructured segments by the inlier rate of the point and line features acquired by the estimated pose using 2P-6DoF. If the inlier rate of lines was higher than the points, 1P1L is utilized in that segment, otherwise, 2P. After that, we counted the success rate on the whole 2017-0828 dataset again over different localization error thresholds and calculate the area under the curve of each method, which is shown in Tab. III. Results show that with the selection mechanism, the performance is further promoted as expected.

VI Conclusions

In this paper, the visual localization problem is reduced to 4DoF with the aid of inertial measurements. Based on this, the unified 2-entity RANSAC pipeline is developed and validated on both synthetic and real data to demonstrate the robustness and efficiency of the methods. Furthermore, we have investigated the proper mechanism to select the appropriate sampling strategy and showed the practicability in the real world application.

Acknowledgment

This work was supported in part by the National Key R&D Program of China (2017YFC0806501), in part by the Science and Technology Project of Zhejiang Province (2019C01043), and in part by the Science and Technology on Space Intelligent Control Laboratory (HTKJ2019KL502002).