Algebraic Characterization of Essential Matrices and Their Averaging in Multiview Settings

TL;DR

This paper introduces a novel algebraic approach for simultaneous estimation of camera positions and orientations in multiview settings, improving accuracy and efficiency in essential matrix averaging for structure from motion.

Contribution

It provides a complete algebraic characterization of conditions for unique Euclidean reconstruction and formulates essential matrix averaging as a constrained optimization problem.

Findings

High accuracy in SfM datasets

Efficient and robust performance

Outperforms existing methods

Abstract

Essential matrix averaging, i.e., the task of recovering camera locations and orientations in calibrated, multiview settings, is a first step in global approaches to Euclidean structure from motion. A common approach to essential matrix averaging is to separately solve for camera orientations and subsequently for camera positions. This paper presents a novel approach that solves simultaneously for both camera orientations and positions. We offer a complete characterization of the algebraic conditions that enable a unique Euclidean reconstruction of $n$ cameras from a collection of $(^n_2)$ essential matrices. We next use these conditions to formulate essential matrix averaging as a constrained optimization problem, allowing us to recover a consistent set of essential matrices given a (possibly partial) set of measured essential matrices computed independently for pairs of images. We finally use the recovered essential matrices to determine the global positions and orientations of the $n$ cameras. We test our method on common SfM datasets, demonstrating high accuracy while maintaining efficiency and robustness, compared to existing methods.

Tables

Table 1. Table 1: Errors in estimated camera orientations, compared to ground truth measurements. R f subscript 𝑅 𝑓 R_{f} denotes the mean Frobenius norm error, averaged over the different cameras, and R d subscript 𝑅 𝑑 R_{d} is the mean angular error in degrees. Empty cells represent missing information.

	Our Method		Chatterjee et al. [5]		Martinec et al. [17]
Data Set	$R_f$	$R_d$	$R_f$	$R_d$	$R_f$	$R_d$
Vienna Cathedral	0.1141	4.7328	0.1514	7.8472	-	-
Piazza del Popolo	0.0595	2.4098	0.2287	11.6022	0.5901	27.2359
NYC Library	0.1200	4.8751	0.1226	6.0765	0.5395	28.4385
Alamo	0.0751	3.0489	0.0879	4.4958	0.1503	7.3180
Metropolis	0.30	15.6613	0.4612	26.5636	1.1381	58.6143
Yorkminster	0.1499	6.6343	0.1526	7.8017	1.3434	73.3679
Montreal ND	0.0608	2.4652	0.1049	5.8742	0.2419	11.928
Tower of London	0.1250	5.0731	0.1366	5.8872	0.3435	16.6457
Ellis Island	0.0636	2.5784	0.0499	2.2486	0.0616	2.4955
Notre Dame	0.0619	2.5091	0.0876	4.7463	0.2109	10.5894

Table 2. Table 2: Camera positions error in meters evaluated on the data sets of [ 25 ] . N c subscript 𝑁 𝑐 N_{c} is the number of images in each dataset, x ¯ , x ~ ¯ 𝑥 ~ 𝑥 \bar{x},\tilde{x} are the mean and median error respectively before bundle adjustment, and x ¯ B ​ A , x ~ B ​ A subscript ¯ 𝑥 𝐵 𝐴 subscript ~ 𝑥 𝐵 𝐴 \bar{x}_{BA},\tilde{x}_{BA} are the mean and median errors after bundle adjustment. N r subscript 𝑁 𝑟 N_{r} are the number of reconstructed cameras. Empty cells represent missing information.

		Our Method					LUD [18]					1DSFM [25]				Cui [6]
Data set	$N_c$	$\overline{x}$	$\tilde{x}$	${\overline{x}}_{BA}$	${\tilde{x}}_{BA}$	$N_r$	$\overline{x}$	$\tilde{x}$	${\overline{x}}_{BA}$	${\tilde{x}}_{BA}$	$N_r$	$\tilde{x}$	${\overline{x}}_{BA}$	${\tilde{x}}_{BA}$	$N_r$	$\tilde{x}$	${\overline{x}}_{BA}$	${\tilde{x}}_{BA}$	$N_r$
Vienna Cathedral	836	9.6	4.2	5.4	1.2	674	10	5.4	10	4.4	750	6.6	2e4	0.5	757	3.5	4.0	2.6	578
Piazza del Popolo	328	7.2	3.5	2.5	0.8	275	5	1.5	4	1.0	305	3.1	200	2.6	303	2.6	3.2	2.4	294
NYC Library	332	3.3	2.2	1.1	0.47	277	6	2.0	7	1.4	320	2.5	20	0.4	292	1.4	6.9	0.9	288
Alamo	577	2.5	1.2	0.8	0.35	482	2	0.4	2	0.3	547	1.1	2e7	0.3	521	0.6	3.7	0.5	500
Metropolis	341	15.2	6.9	2.7	1.4	168	4	1.6	4	1.5	288	9.9	70	1.2	288	-	-	-	-
Yorkminster	437	5.6	2.7	1.9	0.8	341	5	2.7	4	1.3	404	3.4	500	0.2	395	3.7	14	3.8	341
Montreal ND	450	1.9	1.0	0.6	0.4	416	1	0.5	1	0.4	435	2.5	1	0.9	425	0.8	1.1	0.4	426
Tower of London	572	11.6	5.0	4	1.0	414	20	4.7	10	3.3	425	11	40	0.4	414	4.4	6.2	1.1	393
Ellis Island	227	14.1	6.1	5.3	1.7	211	-	-	-	-	-	3.7	40	0.4	213	3.1	1.8	0.6	211
Notre Dame	553	1.8	0.8	0.4	0.2	529	0.8	0.3	0.7	0.2	536	10	7	2.1	500	0.3	0.8	0.2	539

Table 3. Table 3: Runtime in seconds for results in Table 2 . T R + T subscript 𝑇 𝑅 𝑇 T_{R+T} denotes the time for motion averaging (for other methods rotation and translation estimation). T B ​ A subscript 𝑇 𝐵 𝐴 T_{BA} the time for bundle adjustment and T T ​ o ​ t subscript 𝑇 𝑇 𝑜 𝑡 T_{Tot} is the total running time of the method, including the additional time for building the triangle cover. Empty cells represent image collections not tested by the authors. In addition, Cui [ 6 ] does not report results before BA.

	Our Method			LUD [18]			1DSFM [25]			Cui [6]
Data set	$T_{R+T}$	$T_{BA}$	$T_{Tot}$	$T_{R+T}$	$T_{BA}$	$T_{Tot}$	$T_{R+T}$	$T_{BA}$	$T_{Tot}$	$T_{R+T}$	$T_{BA}$	$T_{Tot}$
Vienna Cathedral	68	293	566	787	208	1467	323	3611	3934	-	717	959
Piazza del Popolo	26	27	87	88	31	162	42	213	255	-	93	144
NYC Library	28	58	125	102	47	200	47	382	429	-	48	90
Alamo	47	155	327	385	133	750	152	646	798	-	362	621
Metropolis	16	70	93	67	38	142	47	224	271	-	-	-
Yorkminster	33	116	207	103	148	297	71	955	1026	-	63	108
Montreal ND	41	170	494	271	167	553	93	1043	1136	-	226	351
Tower of London	41	120	241	88	86	228	61	750	811	-	121	221
Ellis Island	21	53	140	-	-	-	29	276	305	-	64	95
Notre Dame	52	277	720	707	126	1047	205	2139	2344	-	793	1159

Equations

$$E_{ij}=R_i^T(T_i-T_j)R_j$$

$$F_{ij}=K_i^{-T}{E}_{ij}{K}_j^{-1}=V_i(T_i-T_j)V_j^T$$

$$\displaystyle V=\left[\begin{array}[]{ccc}R_{1}^{T}\\ \vdots\\ R_{n}^{T}\end{array}\right]\leavevmode\nobreak\ \leavevmode\nobreak\$$

$$E=\left[\hat{U},\hat{V}\right]\left(\begin{array}[]{cc}\Sigma\\ &\Sigma\end{array}\right)\left[\begin{array}[]{c}\hat{V}^{T}\\ \hat{U}^{T}\end{array}\right]$$

$$E=\left[{\hat{U}},{\hat{V}}\right]\left(\begin{array}[]{cc}\Sigma_{+}\\ &\Sigma_{+}\end{array}\right)\left[\begin{array}[]{c}{\hat{V}}^{T}\\ {\hat{U}}^{T}\end{array}\right]$$

$$E=[X,Y]\left(\begin{array}[]{cc}\Sigma_{+}\\ &\Sigma_{-}\end{array}\right)\left[\begin{array}[]{c}X^{T}\\ Y^{T}\end{array}\right]$$

$$A^{T}A=VQD{Q}^T{V}^T.$$

$$Q^T{V}^{T}VQ$$

$$A^{T}A=\left(\frac{1}{\sqrt{n}}V\right)Q(nD)Q^T\left(\frac{1}{\sqrt{n}}{V}^T\right).$$

$$A^{T}A=\hat{V}{\Sigma }^2{\hat{V}}^T.$$

$$E=\left[\hat{U},\hat{V}\right]\left(\begin{array}[]{cc}\Sigma\\ &\Sigma\end{array}\right)\left[\begin{array}[]{c}\hat{V}^{T}\\ \hat{U}^{T}\end{array}\right]$$

$$E=\hat{U}{\Sigma }_{+}{\hat{V}}^T+\hat{V}{\Sigma }_{+}{\hat{U}}^T=\bar{U}{\hat{V}}^T+\hat{V}\overline{U^T},$$

$$E_{ij}$$

$$=\frac{1}{n}{\hat{R}}_i{\hat{T}}_i{\hat{R}}_j^T-\frac{1}{n}{\hat{R}}_i{\hat{T}}_j{\hat{R}}_j^T$$

$$={R_i}^T([{\mathbf{t}}_i{]}_{\times }-[{\mathbf{t}}_j{]}_{\times })R_j,$$

$$E_{ij}$$

$$E=\left[{\hat{U}},{\hat{V}}\right]\left(\begin{array}[]{cc}\Sigma\\ &\Sigma\end{array}\right)\left[\begin{array}[]{c}{\hat{V}}^{T}\\ {\hat{U}}^{T}\end{array}\right]$$

$$E=[X,Y]\left(\begin{array}[]{cc}\Sigma\\ &-\Sigma\end{array}\right)\left[\begin{array}[]{c}X^{T}\\ Y^{T}\end{array}\right]$$

$$\underset{E\in \mathcal{E}}{\mathrm{minimize}}$$

$$rank(E_{\tau (k)})=6$$

$${\Sigma }_{+}(E_{\tau (k)})=-{\Sigma }_{-}(E_{\tau (k)})$$

$$X(E_{\tau (k)})+Y(E_{\tau (k)})\text{ is block rotation,}$$

$$\underset{\Gamma ,\Phi }{\max }\text{leavevmode}\text{leavevmode}\underset{E,B,D}{\min }$$

$$E\in \mathcal{E}$$

$$rank(B_k)=rank(D_k)=6$$

$${\Sigma }_{+}(B_k)=-{\Sigma }_{-}(B_k)$$

$$X(D_k)+Y(D_k)\text{leavevmode}\text{is block rotation}$$

$$L(E_{\tau (k)},B_k,{\Gamma }_k,D_k,{\Phi }_k)={\Vert E_{\tau (k)}-{\hat{E}}_{\tau (k)}\Vert }_F^2+$$

$$\frac{{\alpha }_1}{2}{\Vert B_k-E_{\tau (k)}+{\Gamma }_k\Vert }_F^2+\frac{{\alpha }_2}{2}{\Vert D_k-E_{\tau (k)}+{\Phi }_k\Vert }_F^2.$$

$$B_k^{(0)}={\hat{E}}_{\tau (k)},D_k^{(0)}={\hat{E}}_{\tau (k)},{\Gamma }_k^{(0)}=0,{\Phi }_k^{(0)}=0.$$

$$E^{(t)}=\underset{E}{\mathrm{argmin}}$$

$$+\frac{{\alpha }_1}{2}{\Vert B_k^{(t-1)}-E_{\tau (k)}+{\Gamma }_k^{(t-1)}\Vert }_F^2$$

$$+\frac{{\alpha }_2}{2}{\Vert D_k^{(t-1)}-E_{\tau (k)}+{\Phi }_k^{(t-1)}\Vert }_F^2$$

$$E\in \mathcal{E}$$

$$B_k^{(t)}=\underset{B_k}{\mathrm{argmin}}$$

$$rank(B_k)=6$$

$${\Sigma }_{+}(B_k)=-{\Sigma }_{-}(B_k)$$

$$B_k^{(t)}=U{\Sigma }^{\ast }{U}^T,$$

Full text

00footnotetext: *Equal contributors

1 Introduction

What algebraic conditions make a collection of $n\choose 2$ essential matrices consistent, in the sense that there exist $n$ Euclidean camera matrices that produce them? This fundamental question has not yet been answered in the literature. It is well known that ${3\choose 2}=3$ fundamental matrices are consistent if, and only if, the epipole of the third view is transferred correctly between each pair of views, i.e., for every $1\leq i,j,k\leq 3,$ $\mathbf{e}_{ik}^{T}F_{ij}\mathbf{e}_{jk}=0$. Recent work [15] presented a set of sufficient and necessary algebraic conditions that make $n\choose 2$ fundamental matrices in general position consistent. One could expect that essential matrices that fulfill those same conditions would be consistent with respect to Euclidean camera matrices. However, these conditions are not sufficient and can be contradicted by a counter example, see one such construction in the supplementary material.

Establishing consistency constraints for essential matrices is an important step toward producing essential matrix averaging algorithms. Given $n$ images $I_{1},..,I_{n}$, a common approach for global Structure from Motion (SfM) begins by robustly estimating essential matrices between pairs of views, $\{E_{ij}\}$, from which an estimate of the relative pairwise rotations $\{R_{ij}\}$ and translations $\{\mathbf{t}_{ij}\}$ are extracted. Motion averaging then is performed typically in two steps: first the absolute camera orientations $\{R_{i}\}$ are solved by averaging the relative rotations. Then, using the relative translations and the recovered absolute orientations, the absolute camera positions $\{\mathbf{t}_{i}\}$ are recovered. Finally, the obtained solution is refined by bundle adjustment.

Our goal in this paper is to establish a complete set of necessary and sufficient conditions for the consistency of essential matrices and to use these conditions to formulate a one-step algorithm for averaging essential matrices. To achieve this goal we investigate an object called the $n$-views essential matrix, which is obtained by stacking the $n\choose 2$ essential matrices into a $3n\times 3n$ matrix whose $i,j$’th $3\times 3$ block is the essential matrix $E_{ij}$ relating the $i$’th and the $j$’th frames. We prove that, in addition to projective consistency (introduced in [15]), this matrix must have three pairs of eigenvalues each of the same magnitude but opposite signs, and its eigenvectors directly encode camera parameters.

We use these results to introduce the first (to the best of our knowledge) essential matrix averaging algorithm. Given a noisy estimate of a subset of $n\choose 2$ essential matrices, our algorithm seeks to find the nearest consistent set of essential matrices. We formulate this problem as constrained optimization and solve it using ADMM. We then incorporate this algorithm in a global SfM pipeline and evaluate our pipeline on the datasets of [25], showing superior accuracies relative to state of the art methods on almost all image collections while also maintaining efficiency.

2 Related work

Approaches for Euclidean motion averaging can be divided into two main categories: Incremental methods [16, 20, 1, 14, 26] begin with a small subset of frames and produce an initial reconstruction. The rest of cameras are then used sequentially for reconstruction. These methods are very successful and quite robust. However, they have to apply bundle adjustment refinement at every step to prevent camera drift. Consequently, these method are computationally demanding when applied to large data sets.

Global methods [3, 25, 8, 13, 18], in contrast, recover the motion parameters simultaneously for all the frames. Typical global SfM pipelines proceed by applying a camera orientation solver, followed by a location solver.

Global orientation solvers [3, 17, 24, 9, 5] solve for the absolute orientations of the cameras given relative rotation measurements between pairs of images. [3, 17] derive closed form solutions that minimize a least squares objective constructed from the pairwise relative orientations. These methods are very efficient but due to the relaxed orthonormality requirement, the result is usually suboptimal. Other methods [24, 9, 5] utilize the Lie algebra structure of the rotation group to perform rotation averaging in $SO(3)$. These methods, however, often converge to local minima. Recently, [7] has used the strong duality principle to find the global minimum under certain conditions.

Global location solvers [3, 25, 18] assume known camera orientations and solve for the absolute positions of the cameras by using the noisy relative translations. [3] uses point correspondences to find a least squares solution for the absolute positions. [25] formulates a highly non convex objective and solves for the absolute translations utilizing the Levenberg-Marquet algorithm with random initialization. [18] uses a similar objective while formulating a convex relaxation. [6] uses linear global method that minimizes geometric error in triplet of views while considering feature tracks. All the aforementioned methods are highly dependent on accurate estimation of the absolute rotations of the cameras, which result from a rotation averaging method.

Integrative Methods: [22] formulates the problem of fundamental and essential averaging as global optimization through minimizing the epipolar transfer error. While their method manages to improve the consistency of viewing graphs, it is unable to generate a consistent reconstruction, and so it requires postprocessing steps of both rotation and translation averaging. [19] has introduced the concept of a multi-view essential/fundamental matrix. Their work, however, established only a partial list of constraints. Moreover, their use of a complicated non-convex objective allowed them to only refine a complete reconstruction. [15] has introduced a complete set of consistency conditions for fundamental matrices. They formulate a robust optimization objective and demonstrate state of the art projective reconstructions. Their method, however, is limited to projective settings and is inapplicable to calibrated settings, i.e., for Euclidean reconstruction. [24] suggested a method that optimizes first for camera positions and then for their orientations, and as a post processing simultaneously optimizes for both. However, this method is sensitive to outliers. Recent work explored the properties of the manifold of essential matrices [23]. Their characterization, however, is suitable only for a single essential matrix and not for general multiview settings. Finally, [2, 11] explore general algebraic properties of multi-view settings.

Our paper extends the work of [15, 19] by introducing a complete set of necessary and sufficient conditions for consistency of multiview essential matrices and by proposing an efficient and robust optimization algorithm for essential matrix averaging that incorporates these conditions.

3 Theory

Let $I_{1},...,I_{n}$ denote a collection of $n$ images of a static scene captured respectively by cameras $P_{1},...,P_{n}$. Each camera $P_{i}$ is represented by a $3\times 4$ matrix $P_{i}=K_{i}R_{i}^{T}[I,-t_{i}]$ where $K_{i}$ is a $3\times 3$ calibration matrix, $\bf{t}_{i}\in\mathbb{R}^{3}$ and $R_{i}\in SO(3)$ denote the position and orientation of $P_{i}$, respectively, in some global coordinate system. We further denote $V_{i}=K_{i}^{-T}R_{i}^{T}$, so $P_{i}=V_{i}^{-T}[I,-{\bf t}_{i}]$. Consequently, let $\mathbb{X}=(X,Y,Z)^{T}$ be a scene point in the global coordinate system. Its projection onto $I_{i}$ is given by ${\bf x}_{i}=\mathbb{X}_{i}/Z_{i}$, where $\mathbb{X}_{i}=(X_{i},Y_{i},Z_{i})^{T}=K_{i}R_{i}^{T}(\mathbb{X}-{\bf t}_{i})$.

We denote the fundamental matrix and the essential matrix between images $I_{i}$ and $I_{j}$ by $F_{ij}$ and $E_{ij}$ respectively. It was shown in [3] that $E_{ij}$ and $F_{ij}$ can be written as

[TABLE]

[TABLE]

where $T_{i}=[{\bf t}_{i}]_{\times}$.

Throughout this paper we assume that all calibration matrices are known, so our work deals with solving the problem of Euclidean SfM.

The derivations in this paper adopt the definitions “$n$-view fundamental matrix” and “consistent $n$-view fundamental matrix” from [15]. We first repeat these definitions, for the sake of clarity, and then define analogous definitions for the calibrated case. In the definitions below we denote the space of all the $3n\times 3n$ symmetric matrices by $\mathbb{S}^{3n}$.

Definition 1**.**

A matrix $F\in\mathbb{S}^{3n}$, whose $3\times 3$ blocks are denoted by $F_{ij}$, is called an n-view fundamental matrix if $\forall i\neq j\in\{1,...,n\},$ $rank(F_{ij})=2$ and $\forall i$ $F_{ii}=0$.

Definition 2**.**

An $n$-view fundamental matrix $F$ is called consistent if there exist camera matrices $P_{1},...,P_{n}$ of the form $P_{i}=V_{i}^{-T}[I,\mathbf{t}_{i}]$ such that $F_{ij}=V_{i}([\mathbf{t}_{i}]_{\times}-[\mathbf{t}_{j}]_{\times})V_{j}^{T}$.

Definition 3**.**

A matrix $E\in\mathbb{S}^{3n}$, whose $3\times 3$ blocks are denoted by $E_{ij}$, is called an n-view essential matrix if $\forall i\neq j\in\{1\dots n\}$ , $rank(E_{ij})=2$, the two singular values of $E_{ij}$ are equal, and $\forall i$ $E_{ii}=0$. We denote the set of all such matrices by ${\cal E}$.

Definition 4**.**

An $n$-view essential matrix $E$ is called consistent if there exist $n$ rotation matrices $\{R_{i}\}_{i=1}^{n}$ and $n$ vectors $\{{\bf t}_{i}\}_{i=1}^{n}$ such that $E_{ij}=R_{i}^{T}([\mathbf{t}_{i}]_{\times}-[\mathbf{t}_{j}]_{\times})R_{j}$.

Note that any (consistent) $n$-view essential matrix is also a (consistent) $n$-view fundamental matrix. In [15] necessary and sufficient conditions for the consistency of the $n$-view fundamental matrix were proved. The main theoretical contribution of [15] is summarized in Theorem 1. For the consistency of $n$-view essential matrix, a partial set of necessary conditions were derived in [19]. Those are summarized below in Theorem 2.

Theorem 1**.**

An n-view fundamental matrix $F$ is consistent with a set of $n$ cameras whose centers are not all collinear if, and only if, the following conditions hold:

$Rank(F)=6$* and $F$ has exactly 3 positive and 3 negative eigenvalues.* 2. 2.

$Rank(F_{i})=3$* for all $i=1,...,n$, where $F_{i}$ is the $3\times 3n$ $i^{th}$ block row of $F$.*

Theorem 2**.**

Let $E$ be a consistent $n$-view essential matrix, associated with rotation matrices $\{R_{i}\}_{i=1}^{n}$ and camera centers $\{{\bf t}_{i}\}_{i=1}^{n}$. $E$ satisfies the following conditions

$E$* can be formulated as $E=A+A^{T}$ where $A=UV^{T}$ and $U,V\in\mathbb{R}^{3n\times 3}$*

[TABLE]

with $T_{i}=[{\bf t}_{i}]_{\times}$ and $\sum_{i=1}^{n}{\bf t}_{i}=0$. 2. 2.

Each column of $U$ is orthogonal to each column of $V$, i.e., $V^{T}U=0_{3\times 3}$ 3. 3.

rank(V)=3 4. 4.

If not all $\{{\bf t}_{i}\}_{i=1}^{n}$ are collinear, then rank(U) and rank(A) = 3. Moreover, if the (thin) SVD of $A$ is $A=\hat{U}\Sigma\hat{V^{T}}$, with $\hat{U},\hat{V}\in\mathbb{R}^{3n\times 3}$ and $\Sigma\in\mathbb{R}^{3\times 3}$ then the (thin) SVD of $E$ is

[TABLE]

implying rank(E) = 6.

3.1 Main theoretical results

In this section we derive and prove necessary and sufficient conditions for the consistency of $n$-view essential matrices in terms of their spectral decomposition. These conditions, in turn, will be used later to formulate a constrained optimization problem and to extract the motion parameters from a consistent $n$-view essential matrix $E$ .

Theorem 3**.**

Let $E\in\mathbb{S}^{3n}$ be a consistent $n$-view fundamental matrix with a set of $n$ cameras whose centers are not all collinear. We denote by $\Sigma_{+},\Sigma_{-}\in\mathbb{R}^{3\times 3}$ the diagonal matrices with the 3 positive and 3 negative eigenvalues of $E$, respectively. Then, the following conditions are equivalent:

$E$* is a consistent $n$-view essential matrix* 2. 2.

The (thin) SVD of $E$ can be written in the form

[TABLE]

with $\hat{U},\hat{V}\in\mathbb{R}^{3n\times 3}$ such that each $3\times 3$ block of $\hat{V}$, $\hat{V}_{i}$, $i=1,...,n$, is an $\sqrt{n}$-scaled rotation matrix, i.e., ${\hat{V}_{i}}=\frac{1}{\sqrt{n}}{\hat{R}}_{i}$, where ${\hat{R}}_{i}\in SO(3)$. We say that $\hat{V}$ is a block rotation matrix. 3. 3.

$\Sigma_{+}=-\Sigma_{-}$* and the (thin) spectral decomposition of $E$ is of the form*

[TABLE]

such that $\sqrt{0.5}(X+Y)$ is a block rotation matrix.

Proof.

(1)$\Rightarrow$(2) Assume that $E$ is a consistent $n$-view essential matrix. Then, according to Thm. 2, $E=A+A^{T}$ with $A=UV^{T}$ and $U,V\in\mathbb{R}^{3n\times 3}$ which take the formas in (7). Since $A=UV^{T}$ and $rank(A)=3$, then $A^{T}A=VU^{T}UV$ and $A^{T}A\succeq 0$ with $rank(A^{T}A)=3$ ($A$ and $A^{T}A$ share the same null space). First, we construct a spectral decomposition to $A^{T}A$, relying on the special properties of $U$ and $V$. We have $rank(U)=3$, and therefore $U^{T}U$, which is a $3\times 3$, symmetric positive semi-definite matrix, is of full rank. Its spectral decomposition is of the form $U^{T}U=QDQ^{T}$, where $Q\in SO(3)$. (Spectral decomposition guarantees that $Q\in O(3)$. However, $Q$ can be replaced by $-Q$ if $\det(Q)=-1$.) $D\in\mathbb{R}^{3\times 3}$ is a diagonal matrix consisting of the (positive) eigenvalues of $U^{T}U$. This spectral decomposition yields the following decomposition

[TABLE]

Now, note that

[TABLE]

By a simple manipulation (8) becomes a spectral decomposition

[TABLE]

On the other hand, the (thin) SVD of $A$ is of the form $A={\hat{U}}\Sigma{\hat{V}}^{T}$, where ${\hat{U}},{\hat{V}}\in\mathbb{R}^{3n\times 3}$, $\Sigma\in\mathbb{R}^{3\times 3}$. This means that

[TABLE]

Due to the uniqueness of the eigenvector decomposition, (9) and (10) collapse to the same eigenvector decomposition, up to some global rotation, $H\in SO(3)$, that is $\frac{1}{\sqrt{n}}VQ={\hat{V}}H$, which means that ${\hat{V}}_{i}=\frac{1}{\sqrt{n}}R_{i}^{T}QH^{T}$. Since $R_{i}^{T},Q,H^{T}\in SO(3)$, then ${\hat{R}}_{i}=:R_{i}^{T}QH^{T}\in SO(3)$, showing that ${\hat{V}}$ is a block rotation matrix. Finally, by Thm. 2, the (thin) SVD of $E$ is of the form

[TABLE]

and according to Lemma 5, the eigenvalues of $E$ are $\Sigma$ and $-\Sigma$. Since the elements on the diagonal of $\Sigma$ are positive, and $E$ is symmetric with exactly 3 positive eigenvalues $\Sigma_{+}$ and 3 negative eigenvalues $\Sigma_{-}$, it follows that $\Sigma=\Sigma_{+}$ and $-\Sigma=\Sigma_{-}$ concluding the proof.

(2)$\Rightarrow$(1) Let $E$ be a consistent $n$-view fundamental matrix that satisfies condition (2). We would like to show that $E$ is a consistent $n$-view essential matrix. By condition (2) $E$ can be written as

[TABLE]

where ${\bar{U}}={\hat{U}}\Sigma_{+}$ with ${\hat{V}}_{i}=\frac{1}{\sqrt{n}}{\hat{R}}_{i}$, ${\hat{R}}_{i}\in SO(3)$. By definition $E_{ii}=0$, and this implies that $\bar{U}_{i}\hat{V}_{i}^{T}$ is skew symmetric. Using Lemma 4, $\bar{U}_{i}=\hat{V}_{i}{\hat{T}}_{i}$ for some skew symmetric matrix ${\hat{T}}_{i}=[{\hat{\bf t}}_{i}]_{\times}$. Plugging ${\bar{U}}_{i}$ and ${\hat{V}}_{i}$ in (12) yields

[TABLE]

where $R_{i}={\hat{R}}_{i}^{T}$ and ${\bf t}_{i}=\frac{1}{n}{\hat{\bf t}}_{i}$, concluding the proof.

(2)$\Rightarrow$(3) Let $E$ be an $n$-view fundamental matrix which satisfies condition (2). This means that the (thin) SVD of $E$ can be written in the form E=\left[{\hat{U}},{\hat{V}}\right]\left(\begin{array}[]{cc}\Sigma_{+}\\ &\Sigma_{+}\end{array}\right)\left[\begin{array}[]{c}{\hat{V}}^{T}\\ {\hat{U}}^{T}\end{array}\right], where $\hat{V}$ is a block rotation matrix. Then, by Lemma 5, the (thin) spectral decomposition of $E$ is E=[X,Y]\left(\begin{array}[]{cc}\Sigma_{+}\\ &-\Sigma_{+}\end{array}\right)\left[\begin{array}[]{c}X^{T}\\ Y^{T}\end{array}\right], where $X,Y$ are the eigenvectors of $E$ satisfying  $X=\sqrt{0.5}({\hat{U}}+{\hat{V}})$ and $Y=\sqrt{0.5}({\hat{V}}-{\hat{U}})$. Since ${\hat{V}}=\sqrt{0.5}(X+Y)$, and by condition (2) ${\hat{V}}$ is a block rotation matrix, the claim is confirmed and also $\Sigma_{-}=-\Sigma_{+}$.

(3)$\Rightarrow$(2) Let $E$ be a consistent $n$-view fundamental matrix satisfying condition (3), i.e., its (thin) spectral decomposition is of the form E=\left[{X},{Y}\right]\left(\begin{array}[]{cc}\Sigma_{+}\\ &\Sigma_{-}\end{array}\right)\left[\begin{array}[]{c}{X}^{T}\\ {Y}^{T}\end{array}\right], where $\sqrt{0.5}(X+Y)$ is a block rotation matrix. Since $\Sigma_{+}=-\Sigma_{-}$, we can use Lemma 5, which implies that the (thin) SVD is of the form E=\left[{\hat{U}},{\hat{V}}\right]\left(\begin{array}[]{cc}\Sigma_{+}\\ &\Sigma_{+}\end{array}\right)\left[\begin{array}[]{c}{\hat{V}}^{T}\\ {\hat{U}}^{T}\end{array}\right], where ${\hat{V}}=\sqrt{0.5}(X+Y)$, concluding the proof. ∎

Corollary 1**.**

Euclidean reconstruction. Let $E$ be a consistent $n$-view essential matrix with 6 distinct eigenvalues, then it is possible to explicitly determine $R_{1}\dots R_{n}$ and ${\bf t}_{1},\ldots,{\bf t}_{n}$ that are consistent with respect to $E.$

Proof.

The claim is justified by the following construction, which relies on the spectral characterizations of Thm. 3.

Calculate the eigenvectors $X,Y$ of $E$, and the corresponding three positive eigenvalues, $\Sigma_{+}$, and three negative eigenvalues, $\Sigma_{-}$, respectively. 2. 2.

To realize condition (3) of Thm. 3, there are 8 possible choices of $\sqrt{0.5}(X+YI_{s})$, where I_{s}=\left(\begin{array}[]{ccc}\pm 1&0&0\\ 0&\pm 1&0\\ 0&0&\pm 1\\ \end{array}\right), due to the sign ambiguity of each eigenvector. Then, $I_{s}$ is chosen such that $\sqrt{0.5}(X+YI_{s})$ is block rotation matrix up to a global sign which can be removed. 3. 3.

This spectral decomposition determines directly an SVD decomposition in the form of condition (2) of Thm. 3. We would like to emphasize that due to the multiplicity of singular values, a standard SVD method which is performed directly on $E$, in general, will not produce this special SVD pattern. 4. 4.

Having the relation $E_{ij}=\hat{U}_{i}\Sigma_{+}\hat{V}_{j}^{T}+\hat{V}_{i}\Sigma_{+}\hat{U}_{j}^{T}$ and since $E_{ii}=0$ we get that $\hat{U}_{i}\Sigma_{+}\hat{V}_{i}^{T}$ is skew symmetric. We denote ${\bar{U}}_{i}={\hat{U}}_{i}\Sigma_{+}$ and, by Lemma 4, it holds that ${\hat{T}}_{i}={\hat{V}}_{i}^{-1}{\bar{U}}_{i}$. 5. 5.

Finally, for $i=1,2,..,n$, define $R_{i}=:\sqrt{n}{\hat{V}}_{i}^{T}$ and ${\bf t}_{i}=:\frac{1}{n}{\hat{\bf t}}_{i}$ and it holds that

[TABLE]

∎

This construction yields $\{R_{i}\}_{i=1}^{i=n}$ and $\{{\bf t}_{i}\}_{i=1}^{i=n}$ which are consistent with respect to $E$. Moreover, the reconstruction is unique up to a global similarity transformation. Roughly speaking, this can be proven for $n=3$ by applying an argument from [12]. Next, for $n>3$, by induction, suppose we obtain two reconstructions, $P_{1},...,P_{n}$ and $P^{\prime}_{1},...,P^{\prime}_{n}$. By the induction assumption these must include two sets of $n-1$ non-collinear cameras so that each is unique up to a similarity transformation. Such two sets overlap in at least 2 cameras, which in turn imply that the two similarity transformations must be identical. The complete proof is provided in the supplementary material.

3.2 Supporting lemmas

Lemma 4**.**

[15]** Let $A,B\in\mathbb{R}^{3\times 3}$ with $rank(A)=2$, $rank(B)=3$ and $AB^{T}$ is skew symmetric, then $T=B^{-1}A$ is skew symmetric.

Lemma 5**.**

Let $E\in\mathbb{S}^{3n}$ of rank(6), and $\Sigma\in\mathbb{R}^{3\times 3}$, a diagonal matrix, with positive elements on the diagonal. Let $X,Y,U,V\in\mathbb{R}^{3n\times 3}$, and we define the mapping $(X,Y)\leftrightarrow(U,V):$ $X=\sqrt{0.5}({\hat{U}}+{\hat{V}})$, $Y=\sqrt{0.5}({\hat{V}}-{\hat{U}})$, ${\hat{U}}=\sqrt{0.5}(X-Y),{\hat{V}}=\sqrt{0.5}(X+Y)$.

Then, the (thin) SVD of $E$ is of the form

[TABLE]

if and only if the (thin) spectral decomposition of $E$ is of the form

[TABLE]

Proof.

The proof is provided in the supplementary material. ∎

4 Method

Given images $I_{1},...,I_{n}$, we assume a standard robust method is used to estimate the pairwise essential matrices, which we denote by $\Omega=\{\hat{E}_{ij}\}$. In practice, only a small subset of the pairwise essential matrices are estimated, due to occlusion, large viewpoint and brightness changes as well as objects’ motion, and in addition the available estimates are noisy. Our goal therefore is to find a consistent $n$-view essential matrix $E\in\mathbb{S}^{3n}$ that is as similar to the measured essential matrices as possible.

To make an $n$-view essential matrix consistent, its blocks of pairwise essential matrices must each be scaled by an unknown factor. [19] proposed an optimization scheme that explicitly seeks for the unknown scale factors, yielding a nonlinear, rank-constrained optimization formulation. The success of this approach critically depends on the quality of its initialization, which in experiments was obtained by applying another, state of the art SfM method.

More recently, [15] proposed an analogous approach for projective SfM. They showed that a consistent $3$-view fundamental matrix, which uniquely determines camera matrices (up to a projective ambiguity) from a triplet of images, is invariant to scaling of its pairwise fundamental matrices. This allowed them to formulate an optimization problem that seeks $3$-view fundamental matrices that are both consistent and compatible, while avoiding the need to explicitly optimize for the scale factors.

In this paper, we introduce an optimization scheme that is analogous to that of [15], but adapted to calibrated settings. In particular, our scheme uses the algebraic constraints derived in Thm. 3 to enforce the consistency of noisy, and possibly partial essential matrices. Similar to [15], our method simultaneously enforces consistency of camera triplets attached rigidly to each other, allowing us to avoid optimizing explicitly for the unknown scales of the estimated essential matrices. (To that end we further extend Thm. 3 to handle scaled rotations for image triplets, see supplementary material for details.) Our formulation, however, is more involved than in [15] due to the additional spectral constraints required for Euclidean reconstruction.

In the rest of this section we present our constrained optimization formulation and propose an ADMM-based solution scheme. Subsequently, we discuss how to select minimal subsets of triplets to speed up the optimization. Finally, we show how the results of our optimization can be used to reconstruct the absolute orientations and positions of the $n$ cameras.

4.1 Optimization

In multi-view settings, it is common to define a viewing graph $G=(V,W)$, with nodes $v_{1},\ldots,v_{n}$, corresponding to the $n$ cameras, and $w_{ij}\in W$ if ${\hat{E}}_{ij}$ is one of the estimated pairwise essential matrices. Let $\tau$ denote a collection of $m$ 3-cliques of cameras where $m\leq(^{n}_{3}).$ The collection may be the full set of the 3-cliques in $G$, or a chosen subset as described in Sec. 4.2. We index the elements of $\tau$ by $k=1,...,m$, where $\tau(k)$ denote the $k^{th}$ triplet. The collection $\tau$ determines a partial selection of measured essential matrices, $\Omega$, that plays a role in the optimization problem, where it holds that if $\hat{E}_{ij}\in\Omega$ then $\hat{E}_{ij}^{T}=\hat{E}_{ji}\in\Omega.$

We define the measurements matrix $\hat{E}\in\mathbb{S}^{3n}$ to have $\hat{E}_{ij}$ as its $(i,j)^{th}$ block if $\hat{E}_{ij}\in\Omega$ and $0_{3\times 3}$ in the rest of its blocks. In our optimization problem we look for $E$ that is as close as possible to $\hat{E}$ under the constraints that its $9\times 9$ blocks, induced by $\{\tau(k)\}_{k=1}^{m}$ and denoted by $\{E_{\tau(k)}\}_{k=1}^{m}$, are consistent 3-view essential matrices. In general, such $E$ is inconsistent and incomplete, but as we explain in Sec. 4.3 it is possible to retrieve a set of $n$ absolute rotations and translations that is compatible with its essential matrices up to scale, which in turn implies that the completion of the missing entries is consistent.

We formulate our constrained optimization as follows

[TABLE]

with $i=1,...,n$ and $k=1,...,m$, where $\Sigma_{+}(E_{\tau(k)}),\Sigma_{-}(E_{\tau(k)})\in\mathbb{R}^{3}$ denote the 3 largest (descending order) and $3$ smallest (ascending order) eigenvalues of $E{{}_{\tau(k)}}$ respectively and $X(E_{\tau(k)})\in\mathbb{R}^{9\times 3}$ and $Y(E_{\tau(k)})\in\mathbb{R}^{9\times 3}$ are their corresponding eigenvectors.

Solving (13) is challenging due to its rank and spectral decomposition constraints. We solve this optimization problem using ADMM. To that end, as part of the ADMM method [4] $4m$ auxiliary matrix variables of size $9\times 9$ are added: $2m$ variables duplicating $\{E_{\tau(k)}\}_{k=1}^{m}$, denoted $B=\{B_{k}\}_{k=1}^{m}$ and $D=\{D_{k}\}_{k=1}^{m},$ as well as $2m$ Lagrange multipliers, $\Gamma=\{\Gamma_{k}\}_{k=1}^{m}$ and $\Phi=\{\Phi_{k}\}_{k=1}^{m}$. This yields the following constrained optimization problem

[TABLE]

with $i=1,...,n$ and $k=1,...,m$, where

[TABLE]

We initialize the auxiliary variables at $t=0$ with

[TABLE]

Then, we solve the optimization problem iteratively by alternating between the following four steps.

(i) Solving for $E$.

[TABLE]

This step in the optimization is solved by first neglecting the constraint of two identical singular values for each block, yielding a convex quadratic problem with a closed form solution. Then, based on SVD, each $3\times 3$ block is modified to satisfy the constraint of two identical singular values.

(ii) Solving for $B_{k}$. For all $k=1,...,m$

[TABLE]

The minimizer for this sub-problem is obtained in the following way. By construction, $E_{\tau{(k)}}^{(t)}-\Gamma_{k}^{(t-1)}$ is a symmetric matrix, and we denote its (full) spectral decomposition by $U\Sigma U^{T}$, where $U\in\mathbb{R}^{9\times 9}$ and $\Sigma\in\mathbb{R}^{9\times 9}$ is a diagonal matrix in which the eigenvalues are arranged in a descending order. Then, the update is

[TABLE]

where $\Sigma^{*}\in\mathbb{R}^{9\times 9}$ is a diagonal matrix with the entries

[TABLE]

(iii) Solving for $D_{k}$. For all $k=1,...,m$

[TABLE]

We minimize this sub-problem by an iterative process, which we repeat until convergence. We begin with $D_{k}=E_{\tau(k)}^{(t)}-\Phi_{k}^{(t-1)}$, which is symmetric by construction. We apply spectral decomposition to $D_{k}$, and extract $X(D_{k})$, $Y(D_{k})$, $\Sigma_{+}(D_{k})$ and $\Sigma_{-}(D_{k})$. Assuming no eigenvalue multiplicities, the eigenvectors are determined uniquely up to a sign (this argument is justified in 5.1). We denote by $I_{s}$, a diagonal matrix of size $3\times 3$, such that each diagonal element is either 1 or -1. There are eight configurations for $I_{s}$ from which we select the best, in the sense that on average each $3\times 3$ block, of the form, $\sqrt{0.5}[X+YI_{s}]_{i}$, $i=1,2,3$, is close to scaled rotation, using the following score,

[TABLE]

Next, let $V=\begin{bmatrix}V_{1}\\ V_{2}\\ V_{3}\end{bmatrix}$ be the projection of $\sqrt{0.5}(X+YI_{s}^{*})$ so that $V_{i}$ is the closest scaled rotation to $\sqrt{0.5}[X+YI_{s}^{*}]_{i}$. Projection to scaled $SO(3)$ is obtained through removal of the singular values from the SVD decomposition, setting the scale factor to the average of the singular values, and possibly negating the scale factor to make the determinant positive. Let $U=\sqrt{0.5}(X-YI_{s}^{*})$, and $\tilde{X}=\sqrt{0.5}(U+V)$ and ${\tilde{Y}}=\sqrt{0.5}(V-U)$. We then update the value of $D_{k}$ to be the symmetric matrix

[TABLE]

and repeat these steps until convergence.

(iv) Updating $\Gamma_{k},\Phi_{k}$. For all $k=1,\dots,m$

[TABLE]

4.2 Graph construction and outliers removal

As explained above, to apply our optimization algorithm, it is required to determine a collection $\tau$ of camera triplets, which is a subset of the given camera triplets. The selection of a subset allows for better efficiency and robustness. Similarly to [16, 15], we consider a weighted viewing graph $G$ whose weights for each edge $w_{ij}$ is assigned to be the number of the inlier matches relating $I_{i}$ and $I_{j}$. We begin by selecting disjoint maximal spanning trees from $G$, from which we extract an initial subset of triplets. We then, remove near collinear and inconsistent triplets. We next build a triplet graph $G_{T}$ whose nodes, which represent image triplets, are connected by an edge whenever two triplets share the same two cameras. Finally, we greedily remove nodes from $G_{T}$. starting with the least consistent triplet (using the rotation consistency score defined below), a node is removed as long as the connectivity of $G_{T}$ is preserved and the total number of cameras associated with $G_{T}$ does not decrease.

To define collinear and consistency scores for each triplet we denote the angles in the triangle formed by three cameras $i,j,k$ by $\theta_{i},\theta_{j},\theta_{k}$ respectively. We measure each angle using the known relative translations $\mathbf{t}_{ij},\mathbf{t}_{ik},\mathbf{t}_{jk}$, i.e., $\cos\theta_{i}=(\frac{\mathbf{t}_{ij}^{T}\mathbf{t}_{ik}}{||\mathbf{t}_{ij}||||{\mathbf{t}_{ik}}||})$. Then, the collinearity score of cameras $\{i,j,k\}$ is the minimal angle in $\{\theta_{i},\theta_{j},\theta_{k}\}$. The consistency score of translations is defined by $|\theta_{i}+\theta_{j}+\theta_{k}-\pi|$ and the consistency score of rotations by $||R_{ij}R_{jk}R_{ki}-I||$.

4.3 Location and orientation retrieval

After solving (13), we extract from $E$ the collection of 3-view essential matrices $\{E_{\tau(k)}\}_{k=1}^{m}$, which, due to the optimization, are consistent w.r.t scaled rotations. Next, using Corollary 1 with additional block normalizing at step 3, three rotations $\{R^{\tau(k)}_{1},R^{\tau(k)}_{2},R^{\tau(k)}_{3}\}$ and three translations $\{t^{\tau(k)}_{1},t^{\tau(k)}_{2},t^{\tau(k)}_{3}\}$ are extracted from each $E_{\tau(k)}$, which are uniquely defined up to a similarity transformation. Now, any two triplets in $\tau$ that share two cameras $a,b$ agree on $E_{ab}$. Since the sign of $E_{ab}$ is fixed, it determines the cameras $a,b$ up to 2 unique configurations [10]. Therefore, each one of the two triplets must agree with one of the two configurations. As a result, assuming that both triplets defines the same configuration for $a$ and $b$, there is a unique similarity transformation between the two triplets. In practice, in our experiments we observe that this is always the case.

By the construction process described in Sec. 4.2, the collection of triplets $\tau$ form a connected triplet graph. It is therefore possible to traverse the graph and apply a similarity transformation on the three cameras of each new node $\tau(k),$ and as a result bring all the cameras to a common Euclidean frame.

5 Experiments

To evaluate our approach, we implemented the SfM pipeline described next and tested it on ten challenging collections of unordered internet photographs of various sizes from [25]. Each dataset is provided with ground truth camera parameters. We use our method to recover camera parameters and compare them to the parameters obtained with existing methods, before and after bundle adjustment (BA).

5.1 SfM pipeline

The input to our algorithm is a set of independently estimated pairwise essential matrices, along with the number of inlier matches for each pairwise essential matrix. We build a triplet graph $G_{T}$ as we describe in Sec. 4.2, removing any triplet whose (a) collinearity score is below 0.17 radians, (b) rotation consistency score exceeds 1.1, or (c) translation consistency score exceeds 1 radians. The final connected graph $G_{T}$ defines the collection $\tau$ of triplets of cameras.

Next we apply our optimization algorithm as is described in Sec. 4.1. During optimization we observed that the eigenvalues of $B_{k}$ and $D_{k}$ (16) were always distinct for $k=1,\ldots,m$. This means that the optimization variables, $E_{\tau(k)}$, indeed converge to $3$-view consistent essential matrices with distinct eigenvalues. At this stage, we follow Corollary 1 to recover camera positions and orientations for each triplet of cameras and align all the recovered camera matrices by similarity transformations, as is described in Sec. 4.3. This yields a set of cameras with absolute positions and orientations. The obtained camera parameters are finally refined using BA.

5.2 Results

To evaluate our recovered camera orientations, we compare our results to those obtained with the methods of Chatterjee et al. [5] and Martinec et al. [17]. For a fair comparison we evaluate these methods on same subset of images used in our method. Moreover, since in contrast to our method these solvers do not estimate camera positions we evaluate the results before BA. The results are summarized in Table 1. Our method outperforms these two solvers in nine out of the ten datasets.

To evaluate our recovered camera positions we compare our method to the following position solvers: Cui et al. [6], 1DSFM [25] and LUD [18]. The results are summarized in Table 2. Note that [18, 6] use point matches in their pipelines, while both our method and [25] do not use point correspondences until the final BA. In general, the latter approaches allow for faster optimization, but result in inaccuracies before BA. On the other hand, it allows for greater improvement in the final BA, compared to [18, 6]. Indeed, as can be seen in the table, while our method surpasses all the other methods before bundle adjustment in three out of the ten datasets (according to mean error), it achieves superior performance on eight out of ten after BA.

Table 3 further compares execution times, before and after BA, showing that our method is efficient, compared to existing methods.

5.3 Technical details

For Bundle Adjustment we used the Theia standard SfM library [21]. Camera position results for [18, 6, 25] in Tables 2 and 3 are taken from the papers. We ran our experiments on an Intel(R)-i7 3.60GHz with Windows. Bundle Adjustment was performed on a Linux machine Intel(R) Xeon(R) CPU @ 2.30GHz with 16 cores.

6 Conclusion

We have provided in this paper algebraic conditions for the consistency of essential matrices in multiview settings and an algorithm for their averaging given noisy and partial measurements. In future research we will seek to further incorporate collinear camera triplets in the averaging algorithm, explore numerical properties, and design online consistency enforcement algorithms for SLAM settings.

Acknowledgment This research was supported in part by the Minerva foundation with funding from the Federal German Ministry for Education and Research.

1 Counter example

In the introduction of the paper we claim that a consistent $n$-view fundamental matrix whose $3\times 3$ blocks form essential matrices does not necessarily form a consistent $n$-view essential matrix. We justify this argument by a constructing a counter example for the case $n=3$.

First note that the following observation is true. If ${\bf t}_{2},{\bf t}_{3},{\bf a},{\bf b}\in\mathbb{R}^{3}$ and $R_{2},R_{3}\in SO(3)$ and we set

[TABLE]

and

[TABLE]

then by construction

[TABLE]

is a consistent $n$-view fundamental matrix, unless the outer product ${\bf a}{\bf t}_{3}^{T}$ is such that it reduces the full rank of $V_{3}$ to 2.

Examine each of the block matrices $F_{ij}$:

[TABLE]

It follows from this derivation that $F_{12}$ and $F_{13}$ are essential matrices. Next, we use the identity $[R{\bf t}]_{\times}=R[{\bf t}]_{\times}R^{T}$, which holds for any $R\in SO(3)$ and ${\bf t}\in\mathbb{R}^{3}$, to show that for any ${\bf b}\in\mathbb{R}^{3}$

[TABLE]

The trivial choice of ${\bf a}={\bf b}=0$ yields an essential matrix $F_{23}$ which is consistent with the essential matrices $F_{12}$ and $F_{13}$. Our aim is to make a specific choice of $R_{2},R_{3},{\bf t}_{2},{\bf t}_{3},{\bf a},{\bf b}$ such that $F_{23}$ will be an essential matrix that is inconsistent with the essential matrices $F_{13}$ and $F_{12}$.

1.1 Choosing $R_{2},R_{3},{\bf t}_{2},{\bf t}_{3},{\bf a},{\bf b}$

We first give some intuition for the way we set the values of $R_{2},R_{3},{\bf t}_{2},{\bf t}_{3},{\bf a},{\bf b}$. We look at the term

[TABLE]

and wish to set the values such that this term does not collapse to $R^{T}_{2}R_{3}$, but is still in the form of $R_{2}^{T}R^{*}$, for some $R^{*}\in SO(3)$. To that end, we look for two rotation matrices $R^{*}$ and $R^{**}$ such that $M=R^{*}-R^{**}$ has rank $2$, and set the values such that the term in (22) will be equal to $R_{2}^{T}R^{*}$.

Following the construction of $M$, the SVD of $M$ is of the form

[TABLE]

and we set

[TABLE]

Therefore,

[TABLE]

Now we set $R_{2}$ to be some rotation matrix and $R_{3}=R^{**}$. Clearly,

[TABLE]

which means

[TABLE]

yielding

[TABLE]

which is an essential matrix that in general is inconsistent with $F_{12},F_{13}$.

1.2 Technical details of the code provided for demonstrating a counter example

We first randomly sampled two rotation matrices $R^{*},R^{**}$ until we obtained $M=R^{*}-R^{**}$ of rank $2$, and we set $R_{3}=R^{**}$. Then, we sampled randomly a rotation matrix, and we set $R_{2}$. These selections are stored in a file called “counter_data.mat”.

In the code we assign values to ${\bf t}_{2},{\bf t}_{3},R_{3},{\bf a},{\bf b}$, according to (23). We verify that indeed $F_{12},F_{13}$ and $F_{23}$ are essential matrices. By construction, the $3$-view fundamental matrix $F$ is consistent.

In order to verify that the essential matrices are not consistent we extract the relative rotations from them. Each essential matrix defines two possible relative rotations. We evaluate the relation $R_{12}R_{23}R_{31}$ for each of the 8 choices of triplet of relative rotations, and verify that none of them closes a loop, i.e the following always holds

[TABLE]

An interesting observation. In addition, we verify in the code that for any choice of signs for the eigenvectors of $F$, $X,Y$, it turns out that $\sqrt{0.5}(X+Y)$ is indeed not a block rotation matrix. Interestingly, it holds that $\Sigma_{+}=-\Sigma_{-}$, which shows that by itself this condition, in this case, is not sufficient for defining an appropriate consistent essential matrix. To conclude, requiring from a set of essential matrices to fulfill the sufficient conditions for consistent $n$-view fundamental matrix does not provide sufficient conditions for generating from this set, a consistent essential matrix. The counter example demonstrates the inconsistency of the essential matrices, as well as the violation of our conditions on the consistency of an $n$-view essential matrix.

2 Proof of Lemma 5

Below we prove Lemma 5 from the paper (which we rename here to be Lemma 6).

Lemma 6**.**

Let $E\in\mathbb{S}^{3n}$ of rank(6), and $\Sigma\in\mathbb{R}^{3\times 3}$, a diagonal matrix, with positive elements on the diagonal. Let $X,Y,U,V\in\mathbb{R}^{3n\times 3}$, and we define the mapping $(X,Y)\leftrightarrow(U,V):$ $X=\sqrt{0.5}({\hat{U}}+{\hat{V}})$, $Y=\sqrt{0.5}({\hat{V}}-{\hat{U}})$, ${\hat{U}}=\sqrt{0.5}(X-Y),{\hat{V}}=\sqrt{0.5}(X+Y)$.

Then, the (thin) SVD of $E$ is of the form

[TABLE]

if and only if the (thin) spectral decomposition of $E$ is of the form

[TABLE]

Proof.

($\Rightarrow$)

[TABLE]

where $X=\sqrt{0.5}({\hat{U}}+{\hat{V}})$ and $Y=\sqrt{0.5}({\hat{V}}-{\hat{U}})$. Since, \left[\begin{array}[]{c}\hat{U}^{T}\\ \hat{V}^{T}\end{array}\right]\left[\hat{U},\hat{V}\right]=I_{6\times 6}, it yields \left[\begin{array}[]{c}X^{T}\\ Y^{T}\end{array}\right]\left[X,Y\right]=I_{6\times 6}, concluding that the last term in (28) is indeed (thin) spectral decomposition of $E$.

($\Leftarrow$)

[TABLE]

where ${\hat{U}}=\sqrt{0.5}(X-Y)$ and ${\hat{V}}=\sqrt{0.5}(X+Y)$. The same argument for orthogonality works here, showing that indeed that last term in (39) is SVD of E. ∎

3 Handling scaled rotations

Our optimization enforces the consistency of camera triplets, while allowing the essential matrices to be scaled arbitrarily. This is possible, because our theory can be generalized to handle scaled rotations. Below we generalize Definition 4 from the paper to allow essential matrices of the form $E_{ij}=\alpha_{i}R_{i}^{T}([\mathbf{t}_{i}]_{\times}-[\mathbf{t}_{j}]_{\times})R_{j}\alpha_{j}$ and prove that the main theorem, i.e., Theorem 3 from the paper, holds for this generalization as well. This argument is then used to justify our treatment of camera triplets.

Definition 5**.**

An $n$-view essential matrix $E$ is called scaled consistent if there exist $n$ rotation matrices $\{R_{i}\}_{i=1}^{n}$, $n$ vectors $\{{\bf t}_{i}\}_{i=1}^{n}$ and $n$ non-zero scalars $\{\alpha_{i}\}_{i=1}^{n}$ such that $E_{ij}=\alpha_{i}R_{i}^{T}([\mathbf{t}_{i}]_{\times}-[\mathbf{t}_{j}]_{\times})R_{j}\alpha_{j}$.

The following theorem is a generalized version of Thm. 2 from the paper, and the derivations here are inspired by the derivations made in [19].

Theorem 7**.**

Let $E$ be a scaled consistent $n$-view essential matrix, associated with scaled rotation matrices $\{\alpha_{i}R_{i}\}_{i=1}^{n}$, $\alpha_{i}\neq 0$ and camera centers $\{{\bf t}_{i}\}_{i=1}^{n}$. $E$ satisfies the following conditions

$E$* can be formulated as $E=A+A^{T}$ where $A=UV^{T}$ and $U,V\in\mathbb{R}^{3n\times 3}$*

[TABLE]

with $T_{i}=[{\bf t}_{i}]_{\times}$ and w.l.o.g $\sum_{i=1}^{n}\alpha_{i}^{2}{\bf t}_{i}=0$. 2. 2.

Each column of $U$ is orthogonal to each column of $V$, i.e., $V^{T}U=0_{3\times 3}$ 3. 3.

rank(V)=3 4. 4.

If not all $\{{\bf t}_{i}\}_{i=1}^{n}$ are collinear, then rank(U) and rank(A) = 3. Moreover, if the (thin) SVD of $A$ is $A=\hat{U}\Sigma\hat{V^{T}}$, with $\hat{U},\hat{V}\in\mathbb{R}^{3n\times 3}$ and $\Sigma\in\mathbb{R}^{3\times 3}$ then the (thin) SVD of $E$ is

[TABLE]

implying rank(E) = 6.

Proof.

The decomposition is a straightforward result from Def. 5. Moreover, any global translation of all the camera centers, will not change the values of the entries of $E$. In particular, if we denote the camera centers by $\{{\bf\tilde{t}}_{i}\}_{i=1}^{n}$ and they are translated to their new position ${\bf t}_{i}={\bf\tilde{t}}_{i}-\frac{\sum\alpha_{i}^{2}{\bf\tilde{t}}_{i}}{\sum\alpha_{i}^{2}}$, then $\sum\alpha_{i}^{2}{\bf t}_{i}=0$. 2. 2.

By the decomposition above, we have that

[TABLE]

which concludes that each column of $U$ is orthogonal to each column of $V$. 3. 3.

$rank(V)=3$ since each block of $V$ is of rank 3 and $V$ has 3 columns. 4. 4.

Assume by contradiction that $rank(U)<3$. Then, $\exists{\bf t}\in\mathbb{R}^{3},{\bf t}\neq 0$, s.t. $U{\bf t}=0$. This implies that ${\bf t}_{i}\times{\bf t}=0$ for all $i=1,\ldots,n$. This implies that all the ${\bf t}_{i}$’s are parallel to ${\bf t}$, violating our assumption that not all camera locations are collinear. Consequently $rank(U)=3$ and therefore also $rank(A)=3$. Finally, let $A=\hat{U}\Sigma\hat{V}^{T}$ the SVD of $A$. Since $A=UV^{T}$ we get that

[TABLE]

Then, since $E=A+A^{T}$, we get

[TABLE]

Following the result that the columns of $U$ are orthogonal to those of $V$, it turns out that $\begin{bmatrix}\hat{U}&\hat{V}\end{bmatrix}$ is column orthogonal, concluding that the form above is the SVD of $E$, and $rank(E)=6$.

∎

Next, we show that Thm. 3 in the paper is also applicable with the generalized definition, i.e., Def. 5.

Theorem 8**.**

Let $E\in\mathbb{S}^{3n}$ be a consistent $n$-view fundamental matrix with a set of $n$ cameras whose centers are not all collinear. We denote by $\Sigma_{+},\Sigma_{-}\in\mathbb{R}^{3\times 3}$ the diagonal matrices with the 3 positive and 3 negative eigenvalues of $E$, respectively. The following conditions are equivalent:

$E$* is a scaled consistent $n$-view essential matrix* 2. 2.

The (thin) SVD of $E$ can be written in the form

[TABLE]

with $\hat{U},\hat{V}\in\mathbb{R}^{3n\times 3}$ such that each $3\times 3$ block of $\hat{V}$, $\hat{V}_{i}$, $i=1,\ldots,n$, is a scaled rotation matrix, i.e., ${\hat{V}_{i}}=\hat{\alpha}_{i}{\hat{R}}_{i}$, where ${\hat{R}}_{i}\in SO(3)$ and ${\hat{\alpha}}_{i}\neq 0$. We say that $\hat{V}$ is a scaled block rotation matrix. 3. 3.

$\Sigma_{+}=-\Sigma_{-}$* and the (thin) spectral decomposition of $E$ is of the form*

[TABLE]

such that $\sqrt{0.5}(X+Y)$ is a scaled block rotation matrix.

Proof.

(1)$\Rightarrow$(2) Assume that $E$ is a scaled consistent $n$-view essential matrix. Then, according to Thm. 7, $E=A+A^{T}$ with $A=UV^{T}$ and $U,V\in\mathbb{R}^{3n\times 3}$ with

[TABLE]

where $T_{i}=[{\bf t}_{i}]_{\times}$. Since $A=UV^{T}$ and $rank(A)=3$, then $A^{T}A=VU^{T}UV$ and $A^{T}A\succeq 0$ with $rank(A^{T}A)=3$ ($A$ and $A^{T}A$ share the same null space). First, we construct a spectral decomposition to $A^{T}A$, relying on the special properties of $U$ and $V$. We have $rank(U)=3$, and therefore $U^{T}U$, which is a $3\times 3$, symmetric positive semi-definite matrix, is of full rank. Its spectral decomposition is of the form $U^{T}U=PDP^{T}$, where $P\in SO(3)$. (Spectral decomposition guarantees that $P\in O(3)$. However, $P$ can be replaced by $-P$ if $\det(P)=-1$.) $D\in\mathbb{R}^{3\times 3}$ is a diagonal matrix consisting of the (positive) eigenvalues of $U^{T}U$. This spectral decomposition yields the following decomposition

[TABLE]

Now, note that

[TABLE]

Let $\alpha=\sum_{i=1}^{n}\alpha_{i}^{2}$. By a simple manipulation (48) becomes a spectral decomposition

[TABLE]

On the other hand, the (thin) SVD of $A$ is of the form $A={\hat{U}}\Sigma{\hat{V}}^{T}$, where ${\hat{U}},{\hat{V}}\in\mathbb{R}^{3n\times 3}$, $\Sigma\in\mathbb{R}^{3\times 3}$. This means that

[TABLE]

Due to the uniqueness of the eigenvector decomposition, (49) and (50) collapse to the same eigenvector decomposition, up to some global rotation, $H\in SO(3)$, that is $\frac{1}{\sqrt{\alpha}}VP={\hat{V}}H$, which means that

[TABLE]

Since $R_{i}^{T},P,H^{T}\in SO(3)$, then setting $\hat{\alpha}_{i}=:\frac{\alpha_{i}}{\sqrt{\alpha}}$ and ${\hat{R}}_{i}=:R_{i}^{T}PH^{T}\in SO(3)$, shows that ${\hat{V}}$ is a scaled block rotation matrix. Finally, by Thm. 7, the (thin) SVD of $E$ is of the form

[TABLE]

and according to Lemma 6, the eigenvalues of $E$ are $\Sigma$ and $-\Sigma$. Since the elements on the diagonal of $\Sigma$ are positive, and $E$ is symmetric with exactly 3 positive eigenvalues $\Sigma_{+}$ and 3 negative eigenvalues $\Sigma_{-}$, it follows that $\Sigma=\Sigma_{+}$ and $-\Sigma=\Sigma_{-}$ concluding the proof.

(2)$\Rightarrow$(1) Let $E$ be a consistent $n$-view fundamental matrix that satisfies condition (2). We would like to show that $E$ is a scaled consistent $n$-view essential matrix. By condition (2) $E$ can be written as

[TABLE]

where ${\bar{U}}={\hat{U}}\Sigma_{+}$ with ${\hat{V}}_{i}=\hat{\alpha_{i}}{\hat{R}}_{i}$, ${\hat{R}}_{i}\in SO(3)$. By definition $E_{ii}=0$, and this implies that $\bar{U}_{i}\hat{V}_{i}^{T}$ is a skew symmetric matrix. Using Lemma 4 in the paper, $\bar{U}_{i}=\hat{V}_{i}{\hat{T}}_{i}$ for some skew symmetric matrix ${\hat{T}}_{i}=[{\hat{\bf t}}_{i}]_{\times}$. Plugging ${\bar{U}}_{i}$ and ${\hat{V}}_{i}$ in (53) yields

[TABLE]

where $R_{i}={\hat{R}}_{i}^{T}$, $\alpha_{i}=\hat{\alpha}_{i}$ and ${\bf t}_{i}={\hat{\bf t}}_{i}$, concluding the proof. Finally, the derivation of the equivalence $(2)\Leftrightarrow(3)$ is exactly as in Thm. 3 in the paper. ∎

Corollary 2**.**

Let $E$ be a scaled consistent $n$-view matrix, then

The scale of each block ${\hat{V}}_{i}$ in the scaled block rotation matrix ${\hat{V}}$ can be calculated using by $\hat{\alpha_{i}}=({det({\hat{V}}_{i})})^{\frac{1}{3}}$ 2. 2.

Let $E$ be a scaled consistent $n$-view essential matrix. Then, the transformation $diag(\frac{1}{\hat{\alpha_{1}}}I_{3\times 3},\ldots,\frac{1}{\hat{\alpha_{n}}}I_{3\times 3})\cdot E\cdot diag(\frac{1}{\hat{\alpha_{1}}}I_{3\times 3},\ldots,\frac{1}{\hat{\alpha_{n}}}I_{3\times 3})$ transform $E$ to be a consistent $n$-view essential matrix.

Corollary 3**.**

A scaled consistent 3-view essential matrix is invariant to pairwise scaling.

Proof.

This proof is inspired by the proof of Corollary 2, presented in [15]. Let $E$ be a scaled consistent 3-view essential matrix whose blocks are defined as $E_{ij}=\alpha_{i}R_{i}^{T}(T_{i}-T_{j})R_{j}\alpha_{j}$, and let $\widetilde{E}$ be a $9\times 9$ matrix whose blocks are defined to be $\widetilde{E}_{ij}=s_{ij}E_{ij}$ where $s_{ij}\neq 0$ are arbitrary pairwise scale factors. Without loss of generality we can assume that the number of negative scale factors is even (otherwise we can multiply the entire matrix by -1). Therefore, $s_{1}=(\frac{s_{12}s_{13}}{s_{23}})^{\frac{1}{2}}$, $s_{2}=(\frac{s_{23}s_{12}}{s_{13}})^{\frac{1}{2}}$, and $s_{3}=(\frac{s_{13}s_{23}}{s_{12}})^{\frac{1}{2}}$ determine real values such that $s_{1}s_{2}=s_{12}$, $s_{1}s_{3}=s_{13}$, and $s_{2}s_{3}=s_{23}$. Let $\widetilde{\alpha}_{i}=s_{i}\alpha_{i}$ for $i=1,2,3$, we get that

[TABLE]

Hence, ${\tilde{E}}$ is a scaled consistent 3-view essential matrix. ∎

4 Uniqueness of consistent camera matrices

This part deals with the argument stated in Corollary 1 in the paper, claiming that the recovery of camera matrices from a consistent multiview essential matrix is unique up to a global similarity transformation. More formally, we claim

Theorem 9**.**

Let E be a consistent n-view essential matrix and let $P_{1},\ldots,P_{n}$ be a set of non-collinear camera matrices which is consistent with $E$, then this set of camera matrices is unique up to some global similarity transformation.

We first justify the argument for 3 cameras, exemplify its extension for 4 cameras and finally derives an induction for $n$ cameras.

A calibrated camera matrix $P_{i}$ is represented as

[TABLE]

where $R_{i}$ and ${\bf t}_{i}$ are the orientation and location of the i’th camera, respectively. In addition, we represent a similarity transformation $S(s,R,{\bf t})$ by a matrix of the form

[TABLE]

Then, applying a similarity transformation on a camera matrix $P_{i}$ yields

[TABLE]

which is

[TABLE]

in a calibrated format.

3 cameras. We consider the case of 3 non-collinear cameras. Note that every point correspondence between two of the views can be extended to the third view by intersecting epipolar lines. Consequently, we can produce any number of correspondences across the three views. Uniqueness of reconstruction then follows from [12] who proved that 4 points in three views generally yield unique camera recovery.

4 cameras. We consider a consistent 4-view essential matrix $E$ with 4 corresponding cameras

[TABLE]

Suppose we have another set of cameras which is consistent with $E$,

[TABLE]

We show that these two sets are the same up to a global similarity transformation. Without loss of generality we assume that the third camera is not collinear with the rest of cameras. From the consistency of $E_{123}$, there exists a similarity transformation ${\bar{S}}$ such that

[TABLE]

and from the consistency of $E_{234}$ there exists a similarity transformation ${\hat{S}}$ such that

[TABLE]

Now, since $P_{2}^{*}={\bar{S}}(P_{2})={\hat{S}}(P_{2})$, it yields

[TABLE]

and consequently

[TABLE]

Then we have

[TABLE]

and similarly from the relation $P_{3}^{*}={\bar{S}}(P_{3})={\hat{S}}(P_{3})$ we get that

[TABLE]

By subtracting the last two equations we have

[TABLE]

implying that ${\bar{S}}={\hat{S}}$.

Proof.

The proof is by induction. The argument was proved for 3 cameras and now we suppose that we have a consistent $n$-view essential matrix. From the induction assumption we know that since $E_{1,\ldots,(n-1)}$ is consistent, the recovery of the camera matrices is unique and similarly the recovery of the camera matrices from $E_{2,\ldots,n}$ is unique. As in 4 cameras we assume without loss of generality that the third camera is not collinear with the rest of the cameras, and a similar derivation yields that the recovery of the cameras is unique up to some global similarity transformation. ∎