Globally optimal vertical direction estimation in Atlanta World

TL;DR

This paper introduces a globally optimal method for estimating the vertical direction in Atlanta world scenes, reducing computational complexity by focusing solely on the vertical frame and employing novel bounds for convergence.

Contribution

It proposes a new vertical direction estimation approach using branch-and-bound with innovative bounds, avoiding prior knowledge of horizontal frames and improving efficiency.

Findings

Successfully estimates vertical direction in synthetic and real data

Guarantees global optimality with novel bounds

Handles increased horizontal frames efficiently

Abstract

In man-made environments, such as indoor and urban scenes, most of the objects and structures are organized in the form of orthogonal and parallel planes. These planes can be approximated by the Atlanta world assumption, in which the normals of planes can be represented by the Atlanta frames. Atlanta world assumption, which can be considered as a generalized Manhattan world assumption, has one vertical frame and multiple horizontal frames. Conventionally, given a set of inputs such as surface normals, the Atlanta frame estimation problem can be solved in one-time by branch-and-bound (BnB). However, the runtime of the BnB algorithm will increase greatly when the dimensionality (i.e., the number of horizontal frames) increases. In this paper, we estimate only the vertical direction instead of all Atlanta frames at once. Accordingly, we propose a vertical direction estimation method by considering the relationship between the vertical frame and horizontal frames. Concretely, our approach employs a BnB algorithm to search the vertical direction guaranteeing global optimality without requiring prior knowledge of the number of Atlanta frames. Four novel bounds by mapping 3D-hemisphere to a 2D region are investigated to guarantee convergence. We verify the validity of the proposed method in various challenging synthetic and real-world data.

Tables

Table 1. TABLE I: Different settings for different bounds in Algorithm  1 .

Methods	upper	lower	searching domain
RS	$U^{rot}$	$L^{rot}$	3D cube (side=$2\pi$)
Exp-BnB	$U_q^{exp}$	$L_q^{exp}$	2D square (side=$\pi$)
Ste-circle-BnB	$U_q^{ste}$	$L_q^{ste}$	2D square (side=$2$)
Ste-square-BnB	$U_s^{ste}$	$L_s^{ste}$	2D square (side=$2$)
SCS-BnB	$U_s^{scs}$	$L_s^{scs}$	2D rectangle($2\pi \times \pi /2$)

Table 2. TABLE II: Median runtime and iteration of different methods in NYUv2 data.

Methods	median time(s)	median iteration
Exp-BnB	0.134	816
Ste-circle-BnB	0.184	1010
Ste-square-BnB	4.256	478
SCS-BnB	4.500	675
RANSAC($\rho =0.65$)	0.007	36
RANSAC($\rho =0.75$)	0.013	72
RANSAC($\rho =0.85$)	0.038	203
RANSAC($\rho =0.95$)	0.344	1840

Table 3. TABLE III: Vertical direction estimation results in outdoor data.

Methods	median time(s)	iteration	median error(∘)
Exp-BnB	1.378	239	1.167
Ste-circle-BnB	1.182	223	1.141
Ste-square-BnB	88.470	118	1.141
SCS-BnB	153.356	219	1.129
RANSAC($\rho =0.8$)	0.941	113	1.140
RANSAC($\rho =0.9$)	3.792	459	1.173

Equations

$$\angle (R_a\mathbf{x},R_b\mathbf{x})\le d_{\angle }(R_a,R_b)$$

$$d_{\angle }(R_a,R_b)\le \Vert {\mathbf{r}}_a-{\mathbf{r}}_b\Vert$$

$${\mathbb{S}}_{\ominus }^2=\{\mathbf{x}\in {\mathbb{S}}^2|x_3\ge 0\}$$

$$\max_{\bm{v}\in\mathbb{S}^{2}_{\ominus}}\sum_{j=1}^{N}\mathbbm{1}\Big{(}\mathcal{S}^{\perp}_{j}\vee\mathcal{S}_{j}^{\parallel^{+}}\vee\mathcal{S}_{j}^{\parallel^{-}}\Big{)}$$

$${\mathcal{S}}_j^{{\parallel }^{+}}$$

$${\mathcal{S}}_j^{{\parallel }^{-}}$$

$${\mathcal{S}}_j^{\perp }$$

$$\max_{\bm{v}\in\mathbb{S}^{2}_{\ominus}}\sum_{j=1}^{N}\mathbbm{1}\Big{(}Q_{j}^{\parallel}\vee Q_{j}^{\perp}\Big{)}$$

$$Q_j^{\parallel }$$

$$Q_j^{\perp }$$

$$U_{s}(\mathbb{B})=\sum_{j=1}^{N}\mathbbm{1}\Big{(}\overline{\mathcal{S}}_{j}^{\perp}\vee\overline{\mathcal{S}}_{j}^{\parallel^{+}}\vee\overline{\mathcal{S}}_{j}^{\parallel^{-}}\Big{)}$$

$${\overline{\mathcal{S}}}_j^{{\parallel }^{+}}$$

$${\overline{\mathcal{S}}}_j^{{\parallel }^{-}}$$

$${\overline{\mathcal{S}}}_j^{\perp }$$

$$L_{s}(\mathbb{B})=\sum_{j=1}^{N}\mathbbm{1}\Big{(}\underline{\mathcal{S}}_{j}^{\perp}\vee\underline{\mathcal{S}}_{j}^{\parallel^{+}}\vee\underline{\mathcal{S}}_{j}^{\parallel^{-}}\Big{)}$$

$${\underline{\mathcal{S}}}_j^{{\parallel }^{+}}$$

$${\underline{\mathcal{S}}}_j^{{\parallel }^{-}}$$

$${\underline{\mathcal{S}}}_j^{\perp }$$

$$U_{q}(\mathbb{B})=\sum_{j=1}^{N}\mathbbm{1}\Big{(}\overline{Q}_{j}^{\parallel}\vee\overline{Q}_{j}^{\perp}\Big{)}$$

$${\overline{Q}}_j^{\parallel }$$

$${\overline{Q}}_j^{\perp }$$

$$\lfloor \tau +\psi \rfloor =\{\begin{array}{ll}\tau +\psi ,& \tau +\psi <\pi /2\\ \pi /2,& \tau +\psi \ge \pi /2\end{array}$$

$$L_{q}(\mathbb{B})=\sum_{j=1}^{N}\mathbbm{1}\Big{(}\underline{Q}_{j}^{\parallel}\vee\underline{Q}_{j}^{\perp}\Big{)}$$

$${\underline{Q}}_j^{\parallel }$$

$${\underline{Q}}_j^{\perp }$$

$${\mathbb{S}}_{\ominus }^3=\{\mathbf{q}\in {\mathbb{S}}^3|q_1\ge 0\}$$

$$d_{\angle }(R_a,R_b)=2\angle ({\mathbf{q}}_a,{\mathbf{q}}_b)$$

$$\bm{q}^{T}=\Big{[}\cos(\frac{\|\bm{r}\|}{2}),\sin(\frac{\|\bm{r}\|}{2})\hat{\bm{r}}^{T}\Big{]}$$

$$2\angle ({\mathbf{q}}_a,{\mathbf{q}}_b)\le \Vert {\mathbf{r}}_a-{\mathbf{r}}_b\Vert$$

$$\bm{v}^{T}=\Big{[}\sin(\theta)\hat{\bm{d}}^{T},\cos(\theta)\Big{]}$$

$$\angle ({\mathbf{v}}_a,{\mathbf{v}}_b)\le \Vert {\mathbf{d}}_a-{\mathbf{d}}_b\Vert$$

$$\mathbf{k}$$

$$\mathbf{v}$$

$$h_1$$

$$h_2$$

$$v_1$$

$$v_2$$

Full text

1 Introduction

In man-made environments, scenes usually have structural forms (e.g., the layout of buildings and many indoor objects such as furniture), which can be represented by a set of parallel and orthogonal planes [1]. Atlanta world makes an assumption that the man-made scene can be modeled by a horizontal plane (e.g., ground plane) and many vertical planes (e.g., buildings and walls), then the normals of the planes, which are called world frames, can describe the scenes abstractly. In other words, one vertical frame and multiple horizontal frames could represent Atlanta world [2, 3]. Therefore, it is a crucial step to estimate these vertical and horizontal frame directions in computer vision applications, which is named Atlanta frame estimation [3, 4]. More specifically, structural world frame estimation could be utilized as key modules for various high-level vision applications such as scene understanding [5, 1] and SLAM [6, 7].

Mathematically, an orientation in 3D Euclidean space corresponds to a point in the 3D unit sphere (i.e., $\mathbb{S}^{2}$). This means that the Atlanta frame estimation which estimates multiple orientations is a multiple-clustering (also multi-model fitting) problem in $\mathbb{S}^{2}$. There have been lots of general multiple-clustering algorithms [8, 9, 10] and some of them have been applied in structural world frame estimation [11, 12]. However, Atlanta frame estimation is not exactly the same as the general multiple-clustering problem. It has some special constraints that all horizontal frames are in a plane and the vertical frame is parallel to the normal of the plane. These special constraints reflect essential properties of the Atlanta world assumption. If these constraints are omitted, it will not only lead to a significant decrease in accuracy but also increase the dimensionality of the problem. Furthermore, most of the multiple-clustering algorithms cannot guarantee global optimality when there are lots of outliers in observations [13, 14]. Therefore, recent developments in structural world frame estimation highlight the imminent need for robust and globally optimal methods by considering the above special orthogonal constraints [15, 4].

Recently, Manhattan frame estimation [1], which is a special case of the Atlanta frame estimation, is solved efficiently by a branch-and-bound (BnB) method with the orthogonal constraints [15]. However, when the BnB method is extended to the Atlanta world [3, 4], two problems appear,

The algorithm requires the number of Atlanta frames to be specified, which can rarely be known in advance. Although an automatic method is proposed to estimate the number of horizontal directions in [4], if it is over- or under-estimated, the global optimum may not occur in the correct direction. 2. 2.

It will suffer the curse of dimensionality. There are a considerable number of horizontal directions, whose relationships are unknown which is different from Manhattan world assumption. Consequently, the dimensionality of the problem will increase with the number of horizontal frames, and thus the runtime of the BnB algorithm will increase greatly.

In this paper, we focus on estimating the unique vertical direction instead of all directions in Atlanta world at once. There are two advantages in comparison with the one-time solving all directions methods as follows:

More flexible. The vertical direction is unique in Atlanta frames, and we can estimate the vertical direction even though we don’t know the total number of the horizontal directions. Additionally, we can also estimate the vertical direction from some irregular Atlanta world scenes (e.g., cylindrical buildings in Atlanta, whose horizontal directions number $\rightarrow\infty$). 2. 2.

More efficient. Vertical direction estimation is solved in a closed two dimensional space $\mathbb{S}^{2}$, which is a low-dimensional problem. In other words, only estimating vertical direction can significantly avoid the curse of dimensionality in Atlanta world.

Furthermore, estimating the vertical direction first is always favorable to following operations in practical applications (e.g., scene classification [16], parsing indoor scenes [17] and point set registration [18]). Specially, it is also helpful for estimating other horizontal Atlanta frames, because given the vertical direction, all other horizontal directions will be in a plane, and estimating the other horizontal directions will be a one-dimensional clustering problem in angular space [4]. In other words, given the vertical direction in Atlanta world, it is easy to estimate other horizontal directions with or without knowing the number of horizontal frames  [19, 4].

1.1 Related Work

There is a large body of literature concerned with structural world frame estimation [4, 1, 15, 20]. Since it is a clustering problem in $\mathbb{S}^{2}$ with some orthogonal constraints, we first review the works that apply the classical clustering or fitting method. With the definition of Atlanta world, Expectation Maximization (EM) type algorithms, which are popular for solving the chicken-and-egg problems [21], are applied in direction estimation [2]. However, the EM-type algorithms are local methods and have no guarantee of the global optimality. Therefore, there is an evident risk of local minima, and their performances rely heavily on a good initialization [22]. Besides, the RANdom SAmple Consensus (RANSAC) [23, 24, 25] based multi-structure estimation algorithms (e.g., T-linkage [26] and J-linkage [27]) are applied in structural direction estimation [12, 11]. These RANSAC-type methods are fast, accurate and have the best performances in many cases, but the their solution is sub-optimal due to their obvious heuristic nature [4]. More recently, Straub et al. [1] propose a real-time capable inference algorithm by considering the orthogonal constraints, which uses an adaptive Markov-Chain Monte-Carlo sampling algorithm.

To assure global optimality, J. Bazin et al. propose globally optimal methods [4, 14, 13, 3, 15] by applying branch-and-bound algorithm to solve a consensus set maximization problem. The fundamental theory of these global methods is rotation search [28, 29]. Specifically, the problem is solved by combining Interval Analysis theory with BnB algorithm in [14]. By contrast, the method in [13] is a natural application of Hartley and Kahl’s rotation search theory in $SO(3)$ [28]. Furthermore, 2D-EGI (Extended Gaussian Image) and its integral image are applied in [15] to accelerate the calculation of the bounds in rotation search. Besides, rotation search theory is also extended to Atlanta frame estimation in [3, 4].

However, Atlanta world is more complex than Manhattan world geometrically, since it has more than three frames. Consequently, the globally searching method in [3] requires the number of horizontal directions to be hand-tuned according to the scene, which seems unrealistic in practical applications. Therefore, an automatic two-stage method (meta-BnB) is proposed in [4] to estimate the number of directions. Concretely, it first searches the vertical direction and the horizontal plane in $SO(3)$, then it estimates the horizontal directions in one dimensional angle space. It is worth noting that the meta-BnB is also based on rotation search theory in $SO(3)$. However, searching vertical directions is inherently optimized in $\mathbb{S}^{2}$, whose dimensionality is less than that of $SO(3)$.

Since the rotation search theory is closely related to our work, we then briefly review the rotation search theory in computer vision filed. The rotation search theory has achieved great success in geometric vision problems, for example, point set registration [30, 31], camera calibration [32, 33] and relative pose estimation [34, 28]. Because of the great success of rotation search, there have been several works focusing on improving the efficiency of the algorithm [35, 36, 15, 31].

More specifically, most of the rotation search methods rely on the two following lemmas [28]:

Lemma 1**.**

For $\forall\bm{x}\in\mathbb{S}^{2}$, $R_{a},R_{b}\in SO(3)$, then

[TABLE]

where $d_{\angle}(R_{a},R_{b})$ is the angle lying in the range $[0,\pi]$ of the rotation $R_{a}R_{b}^{-1}$ and $\angle(\cdot,\cdot)$ denotes the angular distance between vectors.

Lemma 2**.**

For $\forall R_{a},R_{b}\in SO(3)$, then

[TABLE]

where $\bm{r}_{a}$ and $\bm{r}_{b}$ are their corresponding angle-axis representations. In Lemma 2, there is a clear indication that the angle distance of two rotations is less than the Euclidean distance in their angle-axis representation. These two lemmas are the basis for the success of the rotation search theory.

Additionally, it is also worth noting that the rotation search usually means optimization in $SO(3)$, which is closely related to $\mathbb{S}^{3}$. Precisely, the homomorphism from a unit quaternion sphere (i.e., $\mathbb{S}^{3}$) to $SO(3)$ is a two-to-one mapping, and then the searching domain $SO(3)$ may be expressed as a hemisphere (including equator) of the unit quaternion sphere [36, 37]. However, the estimation of directions in three-dimensional Euclidean space (i.e., Manhattan or Atlanta frame) is inherently optimized in $\mathbb{S}^{2}$. Unfortunately, there is still a lack of rigid theories regarding globally optimal optimization in $\mathbb{S}^{2}$. In order to estimate the vertical directions in Atlanta world, we originally propose some new and solid mathematical conclusions about searching in $\mathbb{S}^{2}$.

1.2 Our Contribution

In this paper, to overcome the curse of the dimensionality and avoid the difficulty of requiring the user to specify the number of Atlanta frames, we propose a novel method for vertical direction estimation in Atlanta world.The contributions of this work are mainly as follows:

• •

We propose a global searching method for estimating vertical direction, which is different from conventional rotation search in $SO(3)$ [4]. Since the domain of the vertical directions is inherently in $\mathbb{S}^{2}$, then our searching method is more efficient in vertical direction estimation.

• •

Four new different bounds for BnB algorithm are investigated. In contrast to rotation search theory in $SO(3)$, more parametrizations for hemisphere are considered, including exponential mapping, stereographic projection and sphere coordinate system. To the best of our knowledge, it is the first to propose such bounds in $\mathbb{S}^{2}$ to solve structural world frame estimation problem.

2 Methods

2.1 Problem Formulation

In this paper, we estimate the vertical direction from the surface normals in Atlanta world. We denote the input normal set as $\mathcal{N}=\{\bm{n}_{j}\}_{j=1}^{N}$, where $\bm{n}_{j}\in\mathbb{S}^{2}$ is the $j$-th effective unit normal, and $N$ is the number of input normals. In addition, the unknown-but-sought vertical direction is denoted as $\bm{v}$. It is in a hemisphere ($\mathbb{S}^{2}_{\ominus}$), which is defined as:

[TABLE]

where $\bm{x}=[x_{1},x_{2},x_{3}]^{T}$ is a unit vector in $\mathbb{R}^{3}$. Accordingly, the angle of vertical direction and one of the surface normals is lying in range $[0,\pi]$.

To estimate vertical direction robustly, we then apply the inlier maximisation approach to formulate the objective function as

[TABLE]

where $\mathbbm{1}(\cdot)$ is an indicator function which returns 1 if the condition · is true and 0 otherwise and $\vee$ is the logical OR operation. $|\cdot|$ is abs function and $0<\tau<\pi/2$ is the inlier threshold. Eq. (4b), (4c) and (4d) mean that only the surface normals, which are parallel or perpendicular to vertical direction, are inliers. Additionally, because $\bm{n}_{j}^{T}\bm{v}=\cos(\angle(\bm{v},\bm{n}_{j}))$, and when $x\in[0,\pi]$, $\cos(x)$ is a monotonically decreasing function, then an equivalent formulation can be given by

[TABLE]

Since there is no $\arccos$ operation to solve angle in the reformulations, it is more efficient than operating angle inequations.

In rotation search [4], it finds an optimal rotated motion $R$ rather than the optimal direction vector $\bm{v}$ directly. Concretely, given a initial direction vector $\bm{v}_{0}=[0,0,1]^{T}$ and because $R\in SO(3)$, then $\bm{v}=R\bm{v}_{0}\in\mathbb{S}^{2}$. For estimating vertical direction, it is sufficient to search the entire rotation domain and find the optimal $R$ to satisfy that $R\bm{v}_{0}$ is the optimal vertical direction.

2.2 Branch-and-Bound

Finding the optimal $\bm{v}\in\mathbb{S}^{2}_{\ominus}$ to maximize the cardinality of the inlier set is by no mean a trivial problem [38, 39]. Additionaly, the outlier observations, which are unavoidable in the real applications, increase the “hardness” of the estimation problem. Because it is well known that a general robust estimation with outlier observations is an NP-hard problem [40].

To obtain the robust optimal vertical direction, we then use the BnB algorithm. The BnB algorithm is one of the most commonly used tools for solving NP-hard optimization problems, and it is widely applied in many global optimization problems [41]. Briefly, the BnB algorithm recursively divides the searching space into smaller spaces and estimates the upper bound and lower bound of the optimum in each subspace. Then, it removes the sub-spaces which cannot produce a better solution than the best one found so far by the algorithm. The above process is repeated until the best optimum is found within the desired accuracy. The BnB algorithm for estimating vertical direction globally in Atlanta world is outlined in Algorithm 1. It is worth noting that the algorithm only needs the surface normals and the inlier threshold as the inputs without the prior knowledge of the number of horizontal frames.

The key of the BnB algorithm is estimating the upper and lower bounds of the optimum in each subspace tightly and efficiently. In this paper, two general bounds are proposed as follows:

Proposition 1** (General bounds-1).**

Given a branch $\mathbb{B}$, if $\exists\bm{v}_{c}\in\mathbb{B}$, $\forall\bm{v}\in\mathbb{B},\overline{\phi}_{j}\triangleq\max\angle(\bm{v},\bm{n}_{j})$, and $\underline{\phi}_{j}\triangleq\min\angle(\bm{v},\bm{n}_{j})$ then the upper bound can be:

[TABLE]

[TABLE]

[TABLE]

the lower bound can be:

[TABLE]

[TABLE]

Proof.

The rigorous proof can be found in appendix A. ∎

Proposition 2** (General bounds-2).**

Given a branch $\mathbb{B}$, if $\exists\bm{v}_{c}\in\mathbb{B}$, $\forall\bm{v}\in\mathbb{B},\angle(\bm{v},\bm{v}_{c})\leq\max\angle(\bm{v},\bm{v}_{c})\triangleq\psi$, then the upper bound can be:

[TABLE]

[TABLE]

[TABLE]

where

[TABLE]

the lower bound can be:

[TABLE]

[TABLE]

Proof.

The completed proof can be found in appendix B. ∎

Actually, if they have the same $\bm{v}_{c}$ in both general bounds, then $L_{s}=L_{q}$. The main difference between general bounds-1 and general bounds-2 is the calculation of the upper bound. More specifically, given a subspace $\mathbb{B}$, $U_{s}(\mathbb{B})\leq U_{q}(\mathbb{B})$, which means general bounds-1 is tighter than general bounds-2 (Rigorous mathematical proof can be found in appendix C). In the next sections, we introduce how to calculate the upper bound in detail.

2.3 *Parametrizing the Searching Domain *

Before estimating the bounds in BnB algorithm, we must first parametrize the searching space. In this section, we first recall the parametrization of $SO(3)$ in rotation search theory [28, 4], and introduce three different parametrizations of $\mathbb{S}^{2}_{\ominus}$. Furthermore, we analysis the similarities and differences of the parametrizations between $SO(3)$ and $\mathbb{S}^{2}_{\ominus}$.

2.3.1 Parametrization of $SO(3)$

It is well known that rotation space $SO(3)$ can be minimally parametrized with the angle-axis vector, whose norm is the angle of rotation and direction is the axis of the rotation. Therefore, the space of all 3D rotations can be represented by a solid ball of radius ${\pi}$ in $\mathbb{R}^{3}$ [30]. Furthermore, the $\pi$-ball is usually relaxed to a 3D cube for ease of manipulation in the BnB algorithm. Thus Lemma 1 and 2 can be used to efficiently estimate the bounds of rotation search theory.

The Lemma 2 may seem like one of the most fundamental parts in rotation search theory. Let us get down to the details of Lemma 2, and introduce the quaternion to build the connection with the parametrization of $\mathbb{S}^{2}_{\ominus}$. Geometrically, the mapping from quaternions ($\mathbb{S}^{3}$) to rotations ($SO(3)$) is a two-to-one mapping. We then denote a hyper-hemisphere as follows:

[TABLE]

where $\bm{q}=[q_{1},q_{2},q_{3},q_{4}]^{T}$ is a unit vector in $\mathbb{R}^{4}$. Thus the “upper” hemisphere $\mathbb{S}^{3}_{\ominus}$ of the unit quaternion sphere is in one-to-one correspondence with the rotation $\pi$-ball, except at the boundary, where the correspondence is two-to-one [28]. Therefore a conclusion follows as

Lemma 3**.**

For $\forall\bm{q}_{a},\bm{q}_{b}\in\mathbb{S}^{3}_{\ominus}$, then

[TABLE]

where $R_{a},R_{b}\in SO(3)$ are corresponding rotations of $\bm{q}_{a}$ and $\bm{q}_{b}$. This lemma implies that the angle of two rotations is twice the angle between their corresponding quaternions.

Additionally, a unit quaternion $\bm{q}\in\mathbb{S}^{3}_{\ominus}$ can be represented by an angle-axis vector $\bm{r}$ as follows:

[TABLE]

where $\hat{\bm{r}}=\bm{r}/\|\bm{r}\|$ is a unit vector representing the axis of the rotation, and $\|\bm{r}\|$ is the angle of rotation. It is an exponential mapping from the upper quaternion hemisphere to the solid $\pi$-ball. Therefore, there is an important inequation as follows:

Lemma 4**.**

For $\forall\bm{q}_{a},\bm{q}_{b}\in\mathbb{S}^{3}_{\ominus}$, then

[TABLE]

where $\bm{r}_{a}$ and $\bm{r}_{b}$ are angle-axis representations of $\bm{q}_{a}$ and $\bm{q}_{b}$. The complete proofs of Lemma 3 and Lemma 4 can be found in [28] and [37].

According to Eq. (12) and (14), we can easily obtain Lemma 2. In other words, Lemma 2 is separated into two parts, and Lemma 4 inspires us to parametrize the $\mathbb{S}^{2}_{\ominus}$ by exponential mapping.

2.3.2 Parametrization of $\mathbb{S}^{2}_{\ominus}$: Exponential Mapping

Geometrically, $\mathbb{S}^{2}_{\ominus}$ is a hemisphere in three-dimensional Euclidean space, and it is inherently a two-dimensional closed space. In order to parametrize $\mathbb{S}^{2}_{\ominus}$ minimally, we are inspired from Lemma 4 and propose an exponential mapping method to map the hemisphere to a 2D-disk.

Concretely, let $\bm{v}=[v_{1},v_{2},v_{3}]^{T}\in\mathbb{S}^{2}_{\ominus}$, then it can be represented by a corresponding point $\bm{d}\in\mathbb{R}^{2}$ in the disk,

[TABLE]

where $\theta\in[0,\pi/2]$, $\hat{\bm{d}}$ is a unit vector in $\mathbb{R}^{2}$ and $\bm{d}=\theta\hat{\bm{d}}$. Note that the domain of $\theta$ is corresponding to $v_{3}\geqslant 0$, and geometrically, $\theta$ is the radius of the disk. In BnB algorithm, a square (side=$\pi$) circumscribing the mapped disk area is used as the vertical direction domain for ease of manipulation.

The mapping $\bm{v}\rightleftharpoons\bm{d}$ is similar to the mapping from $\mathbb{S}^{3}_{\ominus}$ to the 3D solid $\pi$-ball. Similarly, the mapping is one-to-one except the boundary (i.e., the equator) where it is two-to-one. More specifically, the exponential mapping is closely related to Lie theory [42, 43]. However, in this paper we will not rely on any knowledge of the Lie groups theory without distracting readers’ attention and focus on the direction estimation problem.

Because of the similarity between Eq. (15) and Eq. (13), we then propose a similar inequation as follows:

Proposition 3**.**

For $\forall\bm{v}_{a},\bm{v}_{b}\in\mathbb{S}^{2}_{\ominus}$, then

[TABLE]

where $\bm{d}_{a},\bm{d}_{b}$ are corresponding points of $\bm{v}_{a},\bm{v}_{b}$ in the 2D disk.

Proof.

The complete proof is in the appendix D, and the visualization can be found in Fig. 1. ∎

The exponential parametrization obtains great success in $SO(3)$ and builds the foundation of Lemma 2. In this paper, we extend exponential mapping to $\mathbb{S}^{2}_{\ominus}$ and apply Proposition 3 as one of the fundamental parts in our globally optimal vertical estimation method.

2.3.3 Parametrization of $\mathbb{S}^{2}_{\ominus}$: Stereographic Projection

In geometry, the stereographic projection is a particular mapping that could project a hemisphere to a disk in plane, which means we can also represent the $\mathbb{S}^{2}_{\ominus}$ minimally by applying stereographic projection.

The stereographic projection is described in Fig. 2. We denote a point $\bm{k}=[k_{1},k_{2}]^{T}\in\mathbb{R}^{2}$ in the equatorial plane and its corresponding point $\bm{v}=[v_{1},v_{2}.v_{3}]^{T}\in\mathbb{S}^{2}_{\ominus}$, and if the projection pole is at $[0,0,-1]^{T}$ (South Pole)(see [44]), then we have:

[TABLE]

Clearly, we can parametrize $\mathbb{S}^{2}_{\ominus}$ minimally by stereographic projection. Similarly, a square (side=$2$) circumscribing the mapped disk area is used as the vertical direction domain in the BnB algorithm. It worth noting that the stereographic projection was also applied to accelerate the calculation in rotation search [35], which inspires our work.

2.3.4 Parametrization of $\mathbb{S}^{2}_{\ominus}$: Spherical Coordinate System

The $\mathbb{S}^{2}_{\ominus}$ can also be parameterized by spherical coordinate system (Wikipedia: “spherical coordinate system”). Geometrically, the hemisphere is flattened to a rectangle (see Fig. 3). In the BnB algorithm, the rectangle region can be set as the initial searching domain.

For $\forall\bm{v}=[v_{1},v_{2}.v_{3}]^{T}\in\mathbb{S}^{2}_{\ominus}$ and its corresponding point $\bm{h}=[h_{1},h_{2}]$, the mapping from three-dimensional Cartesian coordinate system to spherical coordinate system is [45]

[TABLE]

where $\arctan(\cdot,\cdot)$ is the four-quadrant inverse tangent function111https://ww2.mathworks.cn/help/matlab/ref/atan2.html. Conversely [46],

[TABLE]

where $-\pi\leq h_{1}\leq\pi$ and $0\leq h_{2}\leq\pi/2$ are azimuth angle and elevation angle, respectively.

In summary, the space of $SO(3)$ is parametrized and relaxed to a 3D cube, thus in BnB algorithm, the cube is recursively divided into eight sub-cubes. By contrast, the parametrizations of $\mathbb{S}^{2}_{\ominus}$ are both disks in exponential mapping and stereographic projection, after that the disk is relaxed to a solid square. In the BnB algorithm, we recursively subdivide it into four smaller squares and calculate the estimation of the upper bound and lower bound for the optimum in each sub-branch. Lastly, $\mathbb{S}^{2}_{\ominus}$ can be parameterized by a azimuth-elevation rectangle using spherical coordinate system. Similarly, in the BnB algorithm, the rectangle is recursively divided into four smaller rectangles.

For ease of understanding, we call the point in the solid disk/rectangle image-point, meanwhile we call its corresponding point preimage-point in $\mathbb{S}^{2}$.

2.4 Estimating Bounds

In this section, we show how to calculate the bounds with different parametrizations in detail.

2.4.1 Bounds of Rotation Search

We first recall the bounds applied in rotation search. According to Lemma 1 and Lemma 2,

[TABLE]

[TABLE]

Then, we have the following Lemma.

Lemma 5** (rotation uncertainty angle bound).**

Given a divided cube-shaped rotation branch $\mathbb{B}^{rot}$, whose center is $\bm{c}^{rot}$, half-side is $\sigma^{rot}$. For $\forall R\in\mathbb{B}^{rot},\forall\bm{v}_{0}\in\mathbb{S}^{2}$,

[TABLE]

where $R_{c}$ is matrix representation of $\bm{c}^{rot}$. Let initial vertical direction $\bm{v}_{0}=[0,0,1]^{T}$, $R\bm{v}_{0}=\bm{v}^{rot}$ and $R_{c}\bm{v}_{0}\triangleq\bm{v}_{c}^{rot}$. Then,

[TABLE]

Observe that it satisfies the conditions of Proposition 2: $\bm{v}^{rot}\rightleftharpoons\bm{v}$, $\bm{v}_{c}^{rot}\rightleftharpoons\bm{v}_{c}$, $\psi^{rot}\rightleftharpoons\psi$ and $\mathbb{B}\rightleftharpoons\{R\bm{v}_{0}|R\in\mathbb{B}^{rot}\}$.

Then, given a divided cube-shaped rotation branch $\mathbb{B}^{rot}$, the bounds can be

[TABLE]

[TABLE]

Note that the bounds are widely used in many geometrical vision problems [4, 15], which are not our original contributions. Besides, it is worth noting that there seems a tighter bound than Eq. (26) in [31], however, to calculate the bound efficiently, it is based on two unproven assumptions.

2.4.2 Bounds Using Exponential Mapping

According to Proposition 3, we have,

Proposition 4**.**

Given a divided square-shaped branch $\mathbb{B}^{exp}$ in exponential mapping plane, whose center is $\bm{d}_{c}^{exp}$, and half-side is $\sigma^{exp}$. For $\forall\bm{d}\in\mathbb{B}^{exp}$,

[TABLE]

where $\bm{v}_{c}^{exp},\bm{v}\in\mathbb{S}^{2}$ are preimage points of $\bm{d}_{c}^{exp}$ and $\bm{d}$.

Proof.

This proposition can be derived as follows:

[TABLE]

which follows Proposition 3 (see Fig. 1). ∎

Proposition 4 and Lemma 5 have similar formulations. However, to the best of our knowledge, it is the first time Proposition 4 has been explicitly introduced to the computer vision field. Obviously, given divided square-shaped branch $\mathbb{B}^{exp}$ in exponential mapping plane, according to Proposition 2, the bounds can be:

[TABLE]

[TABLE]

2.4.3 Bounds Using Stereographic Projection

Stereographic projection has a crucial property that circles are projected as circles (circle preserving [35, 47]). We use this property to calculate the first bound based on stereographic projection.

Proposition 5**.**

Given a divided square-shaped branch $\mathbb{B}^{ste}$ in stereographic projection plane, and its circumscribed circle is $\mathbb{C}^{ste}_{2D}$. The preimage of $\mathbb{C}^{ste}_{2D}$ is $\mathbb{C}^{ste}$ in $\mathbb{S}^{2}$, whose radius is $\sigma^{ste}$ and the direction of its center point is $\bm{v}_{c}^{ste}$; $\forall\bm{k}\in\mathbb{B}^{ste}$, $\bm{v}$ is its preimage-point,

[TABLE]

Proof.

Because $\bm{k}\in\mathbb{B}^{ste}\subset\mathbb{C}^{ste}_{2D}$, then its preimage-point $\bm{v}\in\mathbb{C}^{ste}$. The angle of $\bm{v}$ and $\bm{v}_{c}^{ste}$ must be no greater than the maximum angle $\psi^{ste}$ (see Fig. 4). ∎

We then explain how to calculate $\psi^{ste}$ and $\bm{v}_{c}^{ste}$ in detail. Given a divided square-shaped branch $\mathbb{B}^{ste}$ in mapped plane, its four vertexes ($\bm{k}_{1},\bm{k}_{2},\bm{k}_{3},\bm{k}_{4}$) must be in the edge of circumscribed circle (see Fig. 4). Then the* preimage-points* of the vertexes ($\bm{v}_{1},\bm{v}_{2},\bm{v}_{3},\bm{v}_{4}$) must be in the edge of $\mathbb{C}^{ste}$, and the edge of $\mathbb{C}^{ste}$ is a circle. The direction of the center point $\bm{v}_{c}^{ste}$ is perpendicular to the plane crossing the circle. Hence, $\bm{v}_{c}^{ste}$ is perpendicular to any vector in the circle-plane, which means $\bm{v}_{c}^{ste}\perp(\bm{v}_{1}-\bm{v}_{2})$ and $\bm{v}_{c}^{ste}\perp(\bm{v}_{1}-\bm{v}_{3})$. Let $\bm{v}_{cross}\triangleq$ $(\bm{v}_{1}-\bm{v}_{2})\times(\bm{v}_{1}-\bm{v}_{3})$. Then, $\bm{v}_{c}^{ste}=\bm{v}_{cross}/\|\bm{v}_{cross}\|$ and $\psi^{ste}=\angle(\bm{v}_{c}^{ste},\bm{v}_{1})=\arcsin(\sigma^{ste})$.

Intuitively, Proposition 5 shows that a divided square-shaped branch in the stereographic projection plane is relaxed to a circle, meanwhile the corresponding domain in the 3D sphere is also relaxed to a umbrella-shaped patch surrounded by a circle, whose radius is $\sigma^{ste}$.

Given a divided square-shaped branch $\mathbb{B}^{ste}$ in stereographic projection plane, according to Proposition 2, the bounds can be

[TABLE]

[TABLE]

[TABLE]

The bounds (Eq. (35) and Eq. (36)) are called circle-bounds using stereographic projection as the divided square is relaxed to its circumscribed circle.

2.4.4 Tighter Bounds Using Stereographic Projection

For the stereographic projection, a tighter bound can be found without relaxing the divided square, and therefore, it does not apply the circle-preserving property.

Given a divided square-shaped branch $\mathbb{B}^{ste}$ in stereographic projection plane, the preimage of its center is $\bm{v}_{t}^{ste}$. $\bm{v}\in\mathbb{S}^{2}$ is the preimage-point of $\bm{k}\in\mathbb{B}^{ste}$.

[TABLE]

Considering the Proposition 1, the bounds can be

[TABLE]

[TABLE]

where $\phi_{j}^{ste}=\angle(\bm{v}_{t}^{ste},\bm{n}_{j})$.

The detailed implementation for calculating the $\underline{\phi}_{j}^{ste}$ and $\overline{\phi}_{j}^{ste}$ can be found in appendix E. Note that the bounds are tighter than circle-bounds using stereographic projection. The reason is simple that a divided square is relaxed to a circle in the circle-bounds but no relaxation in the tighter bounds. Additionally, because the bounds are based on the divided square, then we call the tighter bounds square-bounds using stereographic projection.

2.4.5 Bounds Using Sphere Coordinate System

In this part, we introduce the upper and lower bounds using sphere coordinate system according to Proposition 1.

Given a divided rectangle-shaped branch $\mathbb{B}^{scs}$ in azimuth-elevation rectangle, the preimage of its center is $\bm{v}_{c}^{scs}$. $\bm{v}\in\mathbb{S}^{2}$ is the preimage-point of $\bm{h}\in\mathbb{B}^{scs}$.

[TABLE]

Then, the bounds can be

[TABLE]

[TABLE]

where $\phi_{j}^{scs}=\angle(\bm{v}_{c}^{scs},\bm{n}_{j})$.

The implementation for calculating $\underline{\phi}_{j}^{scs}$ and $\overline{\phi}_{j}^{scs}$ can be found in appendix F.

2.4.6 Comparison of the Bounds

It is well known that the success of a BnB algorithm is mainly predicated on the quality of its bounds. To show the relaxation and the tightness, in this section, we compare these bounds (Table I) geometrically.

Bounds of rotation search. The searching domain is parametrized as a 3D cube. In BnB, for each divided sub-cube, it is first relaxed to its circumscribed ball and then relaxed to a region in quaternion sphere (Lemma 2). Lastly, it is relaxed to a spherical patch in $\mathbb{S}^{2}$ using Lemma 1.

Bounds using exponential mapping(exp bounds). Similarly, the searching domain is parametrized as a 2D square. The divided sub-square is first relaxed to its circumscribed circle and then relaxed to a spherical patch in $\mathbb{S}^{2}$ (Proposition 4). Therefore, it has a two-step geometrical relaxation.

Circle-bounds using stereographic projection (ste-cirlce bounds). The searching domain is parametrized as a 2D square. The divided sub-square is first relaxed to its circumscribed circle, which is corresponding to a spherical patch in $\mathbb{S}^{2}$ (circle preserving). In geometric, it has only one relaxation processing.

Square-bounds using stereographic projection (ste-square bounds). The searching domain is the same as that of the ste-circle bounds, however, the ste-square bounds have no geometrical relaxations.

Bounds using sphere coordinate system (SCS bounds). The searching domain is parametrized as a 2D azimuth-elevation rectangle, which leads to significant distortions. Nonetheless, they have no geometrical relaxations.

Note that what we say about geometrical relaxation is only for one specific input. There is a relaxation for the objective, which relaxes the connections among the inputs. In other words, for a large branch, it hardly obtains the upper bound simultaneously for all inputs.

Computational efficiency. The exp-bounds and the ste-circle bounds are calculated more efficiently than the ste-square bounds and the scs-bounds. This is because that to estimate $\overline{\phi}_{j}$ and $\underline{\phi}_{j}$, it is needed to calculating the angle range between the $\bm{n}_{j}$ and four edges of the branch in the ste-square bounds and the scs-bounds. However, given a branch $\mathbb{B}$, all $\{\bm{n}_{j}\}_{j=1}^{N}$ share the same $\psi$ in the exp-bounds and the ste-circle bounds.

3 Experiments

In this section, we verify the validity of the proposed method on challenging synthetic and real-world data. Firstly, we compared our proposed methods with RANSAC and rotation search method to show robustness and efficiency. Then, full Atlanta frame estimation experiments were conducted to verify that estimating vertical direction was helpful to estimating all Atlanta frames. Lastly, we tested proposed methods in two real-world datasets to verify the practicality. All methods were implemented222https://github.com/Liu-Yinlong/Globally-optimal-vertical-direction-estimation-in-Atlanta-world in Matlab 2019a and executed on an AMD Ryzen 7 2700X 3.7GHz CPU.

3.1 Experimental Setting

The settings of approaches/pipelines run on experiments were as follows:

• •

RANSAC: The number of minimal sample subsets was 2. It could get three directions from two inlier-inputs (two inlier directions and its cross product direction), and one of them might be the vertical direction. Besides, the confidence level $\zeta=0.99$ was used for the stopping criterion[23]. The number of iterations was typically taken as

[TABLE]

where $\rho$ was the outlier proportion, $\lceil\cdot\rceil$ returned the nearest integer greater than or equal to the input.

• •

RS: Algorithm 1 with the rotation search bounds. Note that the bounds were also used in meta-BnB in [4]. We did not use the Extended Gaussian Image (EGI) and its integral image [4, 15], because we focused on the geometry and the validity of the proposed bounds. There might be more efficient bounds calculation methods for the proposed bounds but it is out of the scope of this paper.

• •

Exp-BnB: Algorithm 1 with the proposed bounds using exponential mapping.

• •

Ste-circle-BnB: Algorithm 1 with the proposed circle-bounds using stereographic projection.

• •

Ste-square-BnB: Algorithm 1 with the proposed square-bounds using stereographic projection.

• •

SCS-BnB: Algorithm 1 with the proposed bounds using sphere coordinate system.

In addition, to simulate the corrupted inputs in the synthetic experiments, noise and outlier were added. For noise, $\bm{e}_{j}\in\mathbb{R}^{3}$ was the $j$-th random vector, whose elements were randomly uniformly distributed in the interval $[-1,1]$. The noise was simulated by

[TABLE]

where $\kappa$ was the amplitude of noise. For outliers, random orientations were added into the inputs. The total number of inputs was denoted $N$ and the number of outlier inputs was denoted $N_{o}$, then $\rho=N_{o}/N$ was the outlier proportion.

3.2 Synthetic Data Experiments

3.2.1 Synthetic Atlanta World

To simulate synthetic Atlanta world data, a random orientation was generated as the vertical direction ($\bm{v}_{gt}$). Except where otherwise specified, 20% inlier inputs were parallel to vertical direction, and the other 80% inlier inputs were randomly generated to be perpendicular to the vertical direction and thus in the ”horizontal plane”. Note that the number of the horizontal frames were not specified. The inlier threshold was $\tau=\arctan(\kappa)$ according to the noise level in all the synthetic experiments. Once the vertical direction was estimated as $\bm{v}^{*}$, the error was calculated by

[TABLE]

To evaluate the results of the experiment, the vertical error and runtime were recorded. Additionally, because the iteration of the BnB algorithm reflected the tightness of the bounds, the iterations of BnB algorithm with different bounds were also recorded. Moreover, to reduce the randomness, 500 trials were repeated in each setting.

Controlled experiments. We first tested all the methods with different outlier ratios $\rho=\{0.1,\cdots,0.6\}$ and different noise levels $\kappa=\{0.005,0.010,0.020\}$. The number of input was $N=500$. The results are shown in Fig. 5. From the results, we can draw the following conclusions:

• •

All the four types of bounds in $\mathbb{S}^{2}$ and the bounds of rotation search could be nested into the BnB algorithm to estimate the vertical direction globally in Atlanta world.

• •

The four bounds in $\mathbb{S}^{2}$ had different efficiency. Nevertheless, the proposed bounds in $\mathbb{S}^{2}$ were more efficient than the bounds of rotation search.

• •

Broadly, the exp-BnB and the ste-circle-BnB had similar efficiency. The ste-square-BnB and the SCS-BnB had similar efficiency. More specifically, the first two were more efficient than the last two.

• •

Generally, the ste-square-BnB and the SCS-BnB had fewer iterations than the exp-BnB and the ste-circle-BnB. It revealed the ste-square bounds and the SCS bounds were tighter, which was consistent with the previous theoretical analysis.

There are three main reasons why rotation search is rather inefficient for vertical direction estimation.

Multiple solutions. Since $R\bm{v}_{0}=\bm{v^{*}}$, if the initial direction $\bm{v}_{0}$ and the optimal vertical direction $\bm{v^{*}}$ are fixed, there are numerous solutions for $R$ [48]. In other words, if $R\bm{v}_{0}=\bm{v^{*}}$ is correct, then $R_{v}R\bm{v}_{0}=R_{v}\bm{v^{*}}=\bm{v^{*}}$ is also correct, where $R_{v}$ is a rotation about axis $\bm{v^{*}}$. Therefore, all $R_{v}R$ are solutions. For the BnB algorithm, if there are multiple solutions, there are many branches which are very close to one of the solutions and their objective value are also close to the ground truth, then the BnB algorithm must spend lots of time pruning the branches. 2. 2.

Higher dimensionality. Since the vertical direction is inherently a two dimensional problem, searching in higher dimension leads to lower efficiency. 3. 3.

Conservative bound. Since rotation search bounds have a three-step geometrical relaxation, the bounds are relatively conservative.

Furthermore, why exp-BnB and ste-circle-BnB algorithms had more iterations while they still run faster? This was because on one hand, tighter bounds would remove more aggressively and yield fewer iterations. However, on the other hand, using tighter bounds in BnB might be counter-productive if calculating the bound itself took significant time.

Challenging experiments. We conducted more experiments on challenging data. In this part, we only tested the bounds in $\mathbb{S}^{2}$, as the bounds of rotation search were obviously less efficient. The number of input was fixed $N=500$. First, all methods were tested on different high outlier ratios $\rho=\{0.65,\cdots,0.9\}$ and different noise levels $\kappa=\{0.005,0.010,0.020\}$. The results are shown in Fig. 6. Second, all methods were tested on different large noise levels $\kappa=\{0.050,0.100,0.200\}$ and different outlier ratios $\rho=\{0.1,\cdots,0.6\}$. The results are shown in Fig. 7. From the all the results, we can draw the following conclusions:

• •

The exp-BnB had the highest efficiency among all BnB-based methods in such experimental settings. It is worth noting that in large outlier ratio cases ($\rho\geq 0.8$), the exp-BnB algorithm even had comparable efficiency with RANSAC.

• •

The ste-square-BnB had the least iterations among all BnB-based methods, which showed the bounds were very tight.

Theoretically, both ste-square-bounds and SCS-bounds have no geometrical relaxations, then why SCS-bounds needed more iterations than ste-square-bounds in the challenging experiments? This was due to the large distortion of the searching domain. For example, the domain near the optimal direction in $\mathbb{S}^{2}$ might be expanded to a scale-up region in azimuth-elevation rectangle, therefore, the BnB algorithm needed more iterations to prune the near-optimal branches.

3.2.2 Full Atlanta frame estimation

In this part, we verified the performance of our proposed bounds in full Atlanta frame estimation problem. For the sake of fairness, the experiments were conducted on synthetic Manhattan world and the rotation search method was from [13] without EGI-acceleration. Our proposed methods first estimated the vertical frame direction, and then estimated the horizontal frames by a one-dimension clustering method, which can be called sequential methods (see appendix G for more details).

To generate the input normals in Manhattan world, we randomly selected a point $R_{gt}$ in $SO(3)$ as the Manhattan frames. In other words, each column of $R_{gt}$ was corresponding to a Manhattan frame. The experimental settings were $N=500,\kappa=\{0.005,0.010,0.020\}$ and $\rho$ was from 0.1 to 0.6. Once the frame directions had been estimated as $R^{*}$, the estimation error was measured by

[TABLE]

where $mean({\cdot})$ was average function; $\arccos(\cdot)$ was element-wise arccosine function; $max(\cdot)$ was column-wise max function. It computed the average error of the three frames. Note that the solution of rotation search method inherently satisfies the $SO(3)$ constraint, while the solutions of our sequential methods were built without this constraint, as they were formulated for general Atlanta frame estimation.

The results are in Fig. 8. The accuracy of rotation search method was slightly better than that of the sequential methods in large noise level. This was because the rotation search method considered the three orthogonal constraints. Nevertheless, the runtime of rotation search was much longer than that of the sequential methods, due to the fact that the sequential methods had lower dimensionality, tighter bounds and fewer iterations in the BnB framework.

3.3 Real Data Experiments

3.3.1 NYUv2 Data

We tested our method on the NYUv2 Dataset[49], which contained 1449 RGB images, along with the corresponding depths, as well as camera information. The data involved a variety of indoor scenes that were considered to be man-made structural world. In our experiments, we utilized the data to estimate the vertical direction of the scenes. Concretely, we generated the normals from the depth image by the Matlab built-in function pcnormals and estimated the vertical direction from the downsampled normal data ($N\approx 3000$) for all scenes. The threshold was set to $2^{\circ}$ in all methods. For RANSAC, $\rho=\{0.65,0.75,0.85,0.95\}$ were tested since the ground truth of the outlier ratio of each scene was unknown, and the sample iteration was $\Omega$, which was determined by $\rho$ (Eq.(45)).

The distribution of error ($\varepsilon$, see Eq.(47)) is shown in Fig.9. The results revealed that the estimation errors of the BnB algorithms were all concentrated at $0^{\circ}$ and $90^{\circ}$. Because there were some degenerate scenes in the data set, which were degenerated into Manhattan assumption, or even worse, only two main orthogonal frames (Fig.11). Estimating vertical direction in such degenerate scenes might return a frame-direction in horizontal plane. Consequently, some errors were concentrated at $90^{\circ}$. Furthermore, when the outlier proportion $\rho$ was set low, the estimation errors of RANSAC algorithm were not concentrated. When the outlier proportion $\rho$ was set high, the errors were also concentrated. Moreover, to demonstrate the results quantitatively, the $\tau$-recall curve was presented in Fig. 10, where the success case was satisfied $\varepsilon<\tau$ or $\varepsilon>90^{\circ}-\tau$.

Furthermore, the four bounds in $\mathbb{S}^{2}$ had different efficiency. Specifically, the distribution of iteration and runtime in NYUv2 data are in Fig. 12. More specifically, the median runtime and iteration can be found in Table II. Obviously, the exp-BnB algorithm was the most efficient. On the other hand, RANSAC ran very fast when the outlier proportion $\rho$ was set low, however, it might return incorrect results. If the outlier proportion $\rho$ was set high ($\rho=0.95$), its runtime was longer than that of the exp-BnB algorithm. Besides, to compare with rotation search, we directly quote the results from [15]. With rotation search bounds, it needs 117.06s averagely to estimate Manhattan frames for each scene without input sampling. However, with an efficient bounds computation method proposed in [15], it needs only 0.07s on average.

3.3.2 Outdoor Data

In this part, we verified the validity of our methods with the outdoor scene. The data set [50] was recorded in the old town of Bremen, Germany (see Fig.(13)). It contained 13 3D scans, each with up to 22,500,000 points. Estimating the vertical direction first might be useful to register the scenes [18]. For each scene, it was considered as an Atlanta world and $[0,0,-1]^{T}$ was set as the ground truth of vertical direction roughly. We firstly down-sampled the inputs using Matlab built-in function pcdownsample. More specifically, a box grid filter, whose input gridStep was 0.25, was used to reduce the inputs ($N\approx 400,000$). After that their normals were computed by pcnormals, and lastly the vertical direction was estimated from the obtained normals. $\rho=\{0.8,0.9\}$ were set in RANSAC and inlier threshold $\tau=1^{\circ}$ for all methods in this experiments.

The results can be found in Table III (see appendix H for each scene results). Note that the ground truth for vertical direction was roughly set, and the errors were only indicating that the vertical direction estimation results were roughly correct. In this outdoor settings, all bounds in $\mathbb{S}^{2}$ can be nested into the BnB algorithm to globally estimate the vertical direction. Furthermore, the results showed that exp-BnB and ste-circle-BnB algorithm had similar performance and were more efficient. Ste-square-BnB had the least iterations among all the methods, however, it needed more time to calculate the bounds. Furthermore, SCS-BnB algorithm needed much more time to estimate the vertical direction in this experiments. Note that RANSAC could also obtain similar results in this experimental settings. Besides, rotation search method could not terminate in 1800s (30min) in each scene. However, according to the results in [4], with the help of accelerating method, it takes about 80s to estimate Atlanta frames in the whole scene.

4 Conclusion

In this paper, we propose a novel method for estimating the vertical direction in Atlanta world. It can get the globally optimal solution by applying BnB algorithm, without requiring any prior knowledge of the number of frames. Since estimating vertical direction is inherently a two dimensional problem, we propose new bounds in $\mathbb{S}^{2}$ for BnB which are different from the conventional bounds in rotation search.

The experimental results show that all the bounds (in $\mathbb{S}^{2}$ or $SO(3)$) can be nested inside the BnB algorithm to obtain the global solution, and the bounds in $\mathbb{S}^{2}$ outperform the bounds in $SO(3)$, which is the state-of-the-art technique, for estimating vertical direction globally. Furthermore, these four bounds in $\mathbb{S}^{2}$ have different performance. Generally, exp-BnB and ste-circle-BnB have similar performance and are more efficient. Moreover, although ste-square-BnB and SCS-BnB have tighter bounds, they are rather inefficient because of the heavy computational burden. In addition to the quality of the bounds, appropriate parametrization of searching domain is also an important factor of the efficiency of the BnB algorithm. This is why ste-square-BnB is more efficient than SCS-BnB algorithm.

Lastly, since the ste-square-BnB has the least iterations, there may be a hope to accelerate the calculation of the ste-square bounds to obtain a faster BnB algorithm in further work. In addition, since the ste-square bounds are very tight in $\mathbb{S}^{2}$ according to the experimental results, similarly, there may be tighter provable bounds in rotation search ($\mathbb{S}^{3}$) [30, 31].

Acknowledgments

The research leading to these results has partially received funding from the European Unions Horizon 2020 Research and Innovation Program under Grant Agreement No. 785907 (HBP SGA2), from the program of Tongji Hundred Talent Research Professor 2018, and from the Shanghai AI Innovative Development Project 2018. Yinlong, Liu is funded by Chinese Scholarship Council (CSC).