Ideals of the Multiview Variety

TL;DR

This paper investigates the algebraic structure of the multiview variety in computer vision, establishing when certain polynomial sets generate its ideal and clarifying relationships among various proposed ideals.

Contribution

It proves that bifocal and trifocal polynomials generate the multiview ideal under distinct foci and clarifies algebraic relationships among different polynomial ideals in multiview geometry.

Findings

Bifocal and trifocal polynomials generate the multiview ideal with distinct foci.

The multiview ideal is obtained by saturating bifocal polynomials when foci are noncoplanar.

All considered ideals coincide when dehomogenized, describing the space of finite images.

Abstract

The multiview variety of an arrangement of cameras is the Zariski closure of the images of world points in the cameras. The prime vanishing ideal of this complex projective variety is called the multiview ideal. We show that the bifocal and trifocal polynomials from the cameras generate the multiview ideal when the foci are distinct. In the computer vision literature, many sets of (determinantal) polynomials have been proposed to describe the multiview variety. We establish precise algebraic relationships between the multiview ideal and these various ideals. When the camera foci are noncoplanar, we prove that the ideal of bifocal polynomials saturate to give the multiview ideal. Finally, we prove that all the ideals we consider coincide when dehomogenized, to cut out the space of finite images.

Equations

$${\phi }_{\mathcal{A}}:{\mathbb{P}}_{\mathbb{R}}^3⇢({\mathbb{P}}_{\mathbb{R}}^2{)}^n$$

$$\mathcal{A}(p):=\left[\begin{array}{ccccc}{A}_1& p_1& & & \\ A_2& & p_2& & \\ ⋮& & & \ddots & \\ A_n& & & & p_n\end{array}\right].$$

$$minors(4+n,\mathcal{A}(p))\subseteq M_{\mathcal{A}}.$$

$${\mathcal{A}}_{\sigma }(p)=\left(\begin{array}{ccccc}{A}_{{\sigma }_1}& p_{{\sigma }_1}& 0& \dots & 0\\ A_{{\sigma }_2}& 0& p_{{\sigma }_2}& \ddots & 0\\ ⋮& ⋮& \ddots & \ddots & ⋮\\ A_{{\sigma }_k}& 0& \dots & 0& p_{{\sigma }_k}\end{array}\right)$$

$$H_{\mathcal{A}}^k=\sum _{\sigma \in \left(\genfrac{}{}{0ex}{}{[n]}{k}\right)}minors(4+k,{\mathcal{A}}_{\sigma }(p)).$$

$$\left(\begin{array}{ccccccc}{A}_{{\sigma }_1}& p_{{\sigma }_1}& \dots & 0& 0& \dots & 0\\ ⋮& ⋮& \ddots & ⋮& ⋮& \ddots & ⋮\\ A_{{\sigma }_k}& 0& \dots & p_{{\sigma }_k}& 0& \dots & 0\\ (A_{{\tau }_1}{)}_{w_{{\tau }_1}}& 0& \dots & 0& w_{{\tau }_1}& \dots & 0\\ ⋮& ⋮& \ddots & ⋮& ⋮& \ddots & 0\\ (A_{{\tau }_{l-k}}{)}_{w_{{\tau }_{l-k}}}& 0& \dots & 0& 0& \dots & w_{{\tau }_{l-k}}\end{array}\right).$$

$${\chi }_{\mathcal{G}}:\left(\begin{array}{c}{x}_i\\ y_i\\ z_i\end{array}\right)\mapsto G_i^{-1}\left(\begin{array}{c}{x}_i\\ y_i\\ z_i\end{array}\right)$$

$$\det (AB)=\sum _{\sigma \in \left(\genfrac{}{}{0ex}{}{[n]}{m}\right)}\det (A_{[:,\sigma ]})\det (B_{[\sigma ,:]})$$

$${\mathcal{A}}_{[k]}(p)=\left[\begin{array}{ccccc}{A}_1& p_1& & & \\ A_2& & p_2& & \\ ⋮& & & \ddots & \\ A_k& & & & p_k\end{array}\right].$$

$${\mathcal{A}}_{[k]}({\chi }_{\mathcal{G}}(p))=\left[\begin{array}{ccccc}{A}_1& G_1^{-1}{p}_1& & & \\ A_2& & G_2^{-1}{p}_2& & \\ ⋮& & & \ddots & \\ A_k& & & & G_k^{-1}{p}_k\end{array}\right].$$

$$(\mathcal{G}\mathcal{A}{)}_{[k]}(p)=\mathrm{diag}(G_1,\dots ,G_k){\mathcal{A}}_{[k]}(p)$$

$$\left\{\det ({\mathcal{A}}_{[k]}(G^{-1}p{)}_{[\sigma ,:]}):\sigma \in (\genfrac{}{}{0ex}{}{[3k]}{4+k})\right\},$$

$$\det (G_{\tau }$$

$$\sum _{\sigma \in \left(\genfrac{}{}{0ex}{}{[3k]}{4+k}\right)}\det ((G_{\tau }{)}_{[:,\sigma ]})\det ({\mathcal{A}}_{[k]}(G^{-1}p{)}_{[\sigma ,:]}).$$

$${\chi }_{\mathcal{G}}(H_{\mathcal{A}}^k)$$

$$=\sum _{\sigma \in \left(\genfrac{}{}{0ex}{}{[n]}{k}\right)}{H}_{(\mathcal{G}\mathcal{A}{)}_{\sigma }}^k=H_{\mathcal{G}\mathcal{A}}^k.$$

$$f(G_1^{-1}(G_1{A}_1\mathbf{q}),\dots ,G_n^{-1}(G_n{A}_n\mathbf{q}))=0$$

$$H_{\mathcal{A}}^4=H_{\mathcal{G}\mathcal{T}}^4={\chi }_{\mathcal{G}}(H_{\mathcal{T}}^4)\subseteq {\chi }_{\mathcal{G}}(H_{\mathcal{T}}^3)=H_{\mathcal{G}\mathcal{T}}^3=H_{\mathcal{A}}^3.$$

$$M_{\mathcal{A}}={\chi }_{{\mathcal{G}}^{-1}}(M_{\mathcal{G}\mathcal{A}})$$

$$=H_{\mathcal{A}}^2+H_{\mathcal{A}}^3$$

$$H_{\mathcal{A}}^2+H_{\mathcal{A}}^3=M_{\mathcal{A}}\cap ⟨y_4-z_4,y_3-z_3,x_4-z_4,x_3-z_3⟩.$$

$$[p_i{]}_{\times }=\left(\begin{array}{ccc}0& -z_i& y_i\\ z_i& 0& -x_i\\ -y_i& x_i& 0\end{array}\right)$$

$${\mathcal{A}}^F(p):=\left(\begin{array}{c}[p_1{]}_{\times }{A}_1\\ [p_2{]}_{\times }{A}_2\\ ⋮\\ [p_n{]}_{\times }{A}_n\end{array}\right).$$

$${\mathcal{A}}^F(p)=P(p)\mathcal{A}(p)\left[\begin{array}{c}{I}_4\\ 0_{n\times 4}\end{array}\right]=P(p)\mathcal{A}$$

$${\mathcal{A}}^Y(p):=\left[\begin{array}{ccc}{p}_1\times (I{p}_1)& p_1\times 0& \\ p_2\times (B_2{p}_1)& p_2\times t_2& \\ ⋮& ⋮& \\ ⋮& ⋮& \\ p_n\times (B_n{p}_1)& p_n\times t_n& \end{array}\right].$$

$${\mathcal{A}}^Y(p)={\mathcal{A}}^F(p)\left[\begin{array}{cc}{p}_1& 0\\ 0& 1\end{array}\right].$$

$$M_{\mathcal{A}}=⟨y_1{z}_2-y_2{z}_1,x_2{z}_3-x_3{z}_2+y_2{z}_3-y_3{z}_2,$$

$$x_1{z}_3-x_3{z}_1,x_1{x}_3{y}_2+x_1{y}_2{y}_3-x_2{x}_3{y}_1-x_3{y}_1{y}_2⟩.$$

$$H_{\mathcal{A}}^n=M_{\mathcal{A}}$$

$$Y_{\mathcal{A}}=M_{\mathcal{A}}$$

$$\cap ⟨z_1,y_2,x_3,x_2,x_1,z_3^2,z_2{z}_3,z_2^2⟩$$

$$\cap ⟨z_1,y_3,y_2,y_1,x_3,z_3^2,z_2{z}_3,z_2^2⟩,$$

$$F_{\mathcal{A}}=M_{\mathcal{A}}$$

$$\cap ⟨y_3,y_1,x_3,x_1,z_3^2,z_1{z}_3,z_1^2⟩$$

Full text

Pryhuber and Thomas were partially supported by the NSF grant DMS-1719538

1. Introduction

A general projective camera is a rank three matrix in $\mathbb{R}^{3\times 4}$. Given a camera arrangement $\mathcal{A}=(A_{1},\ldots,A_{n})$, the image formation map

[TABLE]

sends a homogenized world point $\mathbf{q}\in\mathbb{P}_{\mathbb{R}}^{3}$ to its images $(\mathbf{p}_{1}=A_{1}\mathbf{q},\ldots,\mathbf{p}_{n}=A_{n}\mathbf{q})\in(\mathbb{P}_{\mathbb{R}}^{2})^{n}$. The $i$th copy of $\mathbb{P}_{\mathbb{R}}^{2}$ in the codomain of $\varphi_{\mathcal{A}}$ is the homogenized image plane of camera $i$. The unique point $\mathbf{c}_{i}\in\mathbb{P}_{\mathbb{R}}^{3}$ in the kernel of $A_{i}$ is the focal point of camera $i$. The map $\varphi_{\mathcal{A}}$ is defined at all points in $\mathbb{P}_{\mathbb{R}}^{3}$ except at the foci $\mathbf{c}_{1},\ldots,\mathbf{c}_{n}$. Triggs called $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$ the joint image [24] and Heyden-Åström call it the natural descriptor [12]. We are interested in studying the complete set of polynomials that vanish on $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$.

Definition 1.1**.**

Given a set $S\subseteq\mathbb{P}_{\mathbb{C}}^{d-1}$, the collection of all polynomials in $\mathbb{C}[x_{1},\ldots,x_{d}]$ that vanish on $S$ is a homogeneous ideal, known as the vanishing ideal of $S$, and denoted as $\mathbf{I}(S)$. The variety $\mathbf{V}(\mathbf{I}(S))$ is the the smallest complex projective variety that contains $S$, known as the Zariski closure of $S$.

We refer the reader to [6] for the basics on ideals and varieties. In this paper we will be interested in the vanishing ideal of the joint image $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$.

Definition 1.2**.**

The multiview ideal of $\mathcal{A}$, denoted $M_{\mathcal{A}}$, is the vanishing ideal of $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$ in $\mathbb{C}[p_{1},\ldots,p_{n}]$ where $p_{i}=(x_{i},y_{i},z_{i})$ are the coordinates on the $i$th copy of $\mathbb{P}^{2}_{\mathbb{C}}$. The Zariski closure of $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$ in $(\mathbb{P}_{\mathbb{C}}^{2})^{n}$ is the complex projective variety $\mathbf{V}(M_{\mathcal{A}})$, which we call the multiview variety of $\mathcal{A}$.

The terminology multiview ideal and multiview variety comes from [2]. Following Triggs [24], Trager et al. refer to the multiview variety as the joint image variety.

Starting with the seminal work of Longuet-Higgins [16], researchers have studied various systems of polynomials that vanish on $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$. In the computer vision literature these equations are known as multiview constraints [19, 7, 11, 17, 12]. Obviously, the ideals generated by these systems of polynomials are contained in $M_{\mathcal{A}}$. However, there hasn’t been much discussion of whether these polynomials generate $M_{\mathcal{A}}$ since the focus of all these papers has been on the multiview variety and not its vanishing ideal. The aim of this paper is to provide a complete description of the multiview ideal and study its relationship to the above sets of polynomials.

It can be difficult to determine the vanishing ideal of a variety. However, there are various advantages to knowing it. To be able to do any computations with a variety or to study its structure using algebra, we need a description in terms of polynomials and the vanishing ideal is the optimal algebraic description. This manifests itself in a number of ways.

The set of all polynomial functions on $X$ is precisely $\mathbb{C}[x_{1},\ldots,x_{d}]/\mathbf{I}(X)$, known as the coordinate ring of $X$. In particular, a polynomial $g$ vanishes on $X$ if and only if $g$ belongs to $\mathbf{I}(X)$. Knowledge of a generating set $\{g_{1},\ldots,g_{k}\}$ of $\mathbf{I}(X)$ also informs us about the local structure of $X$, since a point $x\in X$ is smooth if and only if the Jacobian matrix $(\frac{\partial{g_{i}}}{\partial{x_{j}}})$ has rank equal to the codimension of $X$. More generally, if $X\subset\mathbb{P}_{\mathbb{C}}^{d-1}$ is a projective variety then $\mathbf{I}(X)$ carries all the geometric information about $X$ allowing algebra (and algebraic algorithms) to infer geometric properties of $X$. For example, the dimension and degree of $X$ can be read off from the Hilbert polynomial of $\mathbf{I}(X)$ which also carries many more sophisticated invariants of $X$. See [6] for all the above.

In multiview geometry, many estimation problems can be phrased as polynomial optimization problems over varieties [13, 2]. In particular, the triangulation problem under Gaussian noise amounts to projecting a point onto the multiview variety[1].

In general, polynomial optimization on a variety $X\subseteq\mathbb{R}^{n}$ boils down to certifying the non-negativity of a polynomial $f$ on $X$ by expressing it as a sum-of-squares (sos) modulo an ideal $J$ vanishing on $X$  [3]. This means finding a sos polynomial $s=\sum p_{i}^{2}$ such that $f-s$ lies in $J$. This expressibility is maximized, and the algorithms terminate in the lowest possible degree, when $J=\mathbf{I}(X)$. We illustrate this on a very small example.

Example 1.3**.**

The polynomial $x+1$ is non-negative on $X=\{0\}\subset\mathbb{R}$. The ideal $\langle x^{2}\rangle$ cuts out $X$ but $\mathbf{I}(X)=\langle x\rangle$. Now $(x+1)-1\in\langle x\rangle$ allowing $s=1$ as the sos certificate. On the other hand, if $x+1-s\in\langle x^{2}\rangle$ then $s$ has to have degree at least $2$; for instance $(x+1)-(1+\frac{1}{2}x)^{2}\in\langle x^{2}\rangle$.

The above phenomenon can have a major impact on the number of rounds of convex relaxations needed to solve a polynomial optimization problem such as the well-known Lasserre/sos hierarchies [14, 20], where each round looks for sos certificates of a fixed degree with degrees increasing monotonically with rounds. In each round the semidefinite program being solved is of size $O(n^{d})$, where $n$ is the number of variables and $d$ is degree in that round. As a result, in many cases only the first round maybe computationally feasible and having access to $\mathbf{I}(X)$ can make the difference between the problem being tractable or not.

The rest of the paper is structured as follows. After a brief discussion of the notation used in this paper we begin in Section 2 by introducing a family of ideals associated with every camera arrangement $\mathcal{A}$ which we call the $k$-focal ideals. We describe how these ideals behave under change of coordinates, and dispel the popular myth that, under a change of image coordinates, $k$-focal polynomials go to $k$-focal polynomials. In Section 3, we prove our first main theorem (Theorem 3.7), that the well-known bifocal (epipolar constraints) and trifocal polynomials generate $M_{\mathcal{A}}$ when the camera foci in $\mathcal{A}$ are distinct. Next, in Section 4, we consider three different types of determinantal polynomials proposed to cut out the multiview variety by Heyden-Åström [12], Faugeras et al.  [7] and Ma et al.  [17]. We show that while the ideals they generate are all contained in $M_{\mathcal{A}}$, none of them actually coincide with $M_{\mathcal{A}}$. We establish their precise algebraic relationship with $M_{\mathcal{A}}$. In Section 5, we consider the relationship of the multiview ideal to bifocal polynomials and prove the algebraic analog of the statement that the bifocal polynomials cut out the multiview variety when the camera foci are noncoplanar. In Section 6, we study how the various ideals relate to each other when we restrict our attention to finite images, i.e. exclude points at infinity. We conclude in Section 7 with a summary.

Many results in this paper require explicit computation. We recommend the reader have a copy of Macaulay2 [9] (or equivalent symbolic algebra software) handy. The Macaulay2 codes for our computations can be found at https://sites.math.washington.edu/~thomas/papers/Multiview_Ideal.zip

1.1. Notation

In the rest of the paper, we will use $\mathbb{P}$ to denote $\mathbb{P}_{\mathbb{C}}$. The ideal generated by the polynomials $f_{1},\ldots,f_{s}$ will be denoted as $\langle f_{1},\ldots,f_{s}\rangle$.

We will use $A$ for cameras and $G$ for matrices in $\textup{GL}_{n}$. $\mathcal{A}$ and $\mathcal{G}$ will denote arrangements of corresponding matrices. Bold, lower-case roman letters will be used to indicate vectors, and lower-case greek letters will be used for functions. Given a partial symbolic matrix $M$, $minors(k,M)$ will denote the ideal generated by all $k\times k$ minors of the matrix $M$. The symbol $[n]$ denotes the set $\{1,\ldots,n\}$ and $\binom{[n]}{m}$ denotes the set of all size $m$ subsets of $[n]$.

2. The $k$-focal ideals of a camera arrangement

Let $p_{i}$ be the tuple of variables $(x_{i},y_{i},z_{i})$ denoting the coordinates associated to the projective plane $\mathbb{P}_{\mathbb{R}}^{2}$ corresponding to the $i$th camera image. Write $p=(p_{1},\ldots,p_{n})$, and consider the partially symbolic matrix

[TABLE]

Let $\mathcal{A}(\mathbf{p})$ denote the evaluation of $\mathcal{A}(p)$ at $p=\mathbf{p}$. If $\mathbf{p}:=(\mathbf{p}_{1},\ldots,\mathbf{p}_{n})\in\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$ then there exists some $\mathbf{q}\in\mathbb{P}_{\mathbb{R}}^{3}$ and scalars $\lambda_{i}\in\mathbb{R}$ such that $A_{i}\mathbf{q}=\lambda_{i}\mathbf{p}_{i}$ for all $i=1,\ldots,n$. Therefore, $\mathcal{A}(\mathbf{p})$ has a non-trivial kernel since it contains the point $(\mathbf{q},-\lambda_{1},\ldots,-\lambda_{n})$, and hence the maximal minors of $\mathcal{A}(p)$, which are polynomials in $p_{1},\ldots,p_{n}$, vanish on $\mathbf{p}$. Since $\mathbf{p}$ was arbitrary, these maximal minors vanish on all of $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$ and on the multiview variety. Therefore,

[TABLE]

In this section, we describe further minors of $\mathcal{A}(p)$ and the ideals they generate, which will play an important role in the description of $M_{\mathcal{A}}$.

Definition 2.1**.**

For a subset $\sigma=\{\sigma_{1},\dots,\sigma_{k}\}\subseteq[n]$ where $k\geq 2$, consider the partially symbolic matrix

[TABLE]

of size $3k\times(4+k)$. A maximal $(4+k)\times(4+k)$ minor of $\mathcal{A}_{\sigma}(p)$ is called a $k$-focal polynomial* of $\mathcal{A}$. The $k$-focal ideal of $\mathcal{A}$, $H_{\mathcal{A}}^{k}$, is the ideal sum*

[TABLE]

Trager et al. also study the $k$-focal polynomials and refer to them as $k$-linearities [21, 22]. Note that every $k$-focal polynomial is multilinear and of total degree $k$. Such a minor involves choosing $4+k$ rows of $\mathcal{A}_{\sigma}(p)$, and by a pigeonhole argument, at most four cameras may contribute more than one row to the minor when $k>4$. Indeed, if more than four cameras contributed at least two rows each, then at least $10$ rows are accounted for, which leaves at most $k-6$ rows to take from the remaining $k-5$ cameras. So at least one camera will be left out entirely which means that the submatrix of that $4+k$ minor has a zero column and the minor is zero.

A useful fact for us will be that for two positive integers $l>k\geq 2$, there is a simple way to “bump up” a $k$-focal polynomial to an $l$-focal polynomial by multiplying the $k$-focal polynomial with a monomial.

Lemma 2.2**.**

Suppose $f$ is a $k$-focal polynomial from cameras $\sigma=\{\sigma_{1},\dots,\sigma_{k}\}\subset[n]$ where $k\geq 2$. For any $l>k$ cameras $\tau=\{\sigma_{1},\dots,\sigma_{k},\tau_{1},\dots,\tau_{l-k}\}$, there is a $l$-focal polynomial $g$ such that $(\prod_{i=1}^{l-k}w_{\tau_{i}})f=g$ for any choice of variables $w_{\tau_{i}}\in\{x_{\tau_{i}},y_{\tau_{i}},z_{\tau_{i}}\}$, one for each camera.

Proof.

Add the row and column associated to coordinate $w_{\tau_{i}}$ to ${\mathcal{A}_{\sigma}}(p)$ for $\tau_{1},\dots,\tau_{l-k}$ as follows

[TABLE]

Taking the determinant of this matrix yields the $l$-focal polynomial $g=(\prod_{i=1}^{l-k}w_{\tau_{i}})f$. ∎

Combining the above facts we get that any $l$-focal polynomial for $l>4$ is of the form $(\prod_{i=1}^{l-k}w_{\tau_{i}})f$ where $f$ is a $k\leq 4$ focal polynomial. This is a generalization of Proposition 2 in [21] that showed that every $n$-focal polynomial is a monomial multiple of a $k$-focal polynomial for $k\leq 4$. As a result, we will primarily focus on the ideals $H_{\mathcal{A}}^{2}$, $H_{\mathcal{A}}^{3}$, and $H_{\mathcal{A}}^{4}$, called the bifocal, trifocal, and quadrifocal ideals of $\mathcal{A}$.

A closer look at $H^{2}_{\mathcal{A}}$ reveals that it is the ideal generated by the $n\choose 2$ epipolar constraints, since $\mathcal{A}_{\{i,j\}}$ is a $6\times 6$ matrix, whose determinant is the epipolar constraint between images $i$ and $j$. By Lemma 2.2, $H^{3}_{\mathcal{A}}$ contains the bumped up version of $H^{2}_{\mathcal{A}}$ and for every triplet of images $\{i,j,k\}$, the 27 trifocals implied by the three trifocal tensors relating them. And finally, $H^{4}_{\mathcal{A}}$ contains the bumped up versions of $H^{2}_{\mathcal{A}}$ and $H^{3}_{\mathcal{A}}$ and the 81 quadrifocals implied by the quadrifocal tensor. The fact that we only need to study $H_{\mathcal{A}}^{2}$, $H_{\mathcal{A}}^{3}$, and $H_{\mathcal{A}}^{4}$ lines up with the well known fact in multiview geometry that when studying $n$-view constraints, one only needs to study the epipolar matrix, the trifocal tensor and the quadrifocal tensor. See Chapter 17 in the book by Hartley & Zisserman [11] for explicit computations of the generators of $H^{2}_{\mathcal{A}},H^{3}_{\mathcal{A}},$ and $H^{4}_{\mathcal{A}}$ and their history.

In the remainder of this section, we will investigate how $k$-focal ideals transform under certain linear transformations on cameras. It is widely known that, from image data, the geometry of a camera arrangement can only be determined up to an arbitrary choice of $\mathbb{P}^{3}$ coordinates. This is reflected in the following lemma.

Lemma 2.3** (Projective Ambiguity).**

Suppose $G\in\operatorname{GL}_{4}$. Then for any $k$, $H_{\mathcal{A}}^{k}=H^{k}_{\mathcal{A}G}$ where $\mathcal{A}G=(A_{1}G,A_{2}G,\ldots,A_{k}G)$.

Proof.

This follows since $(\mathcal{A}G)_{\sigma}(p)=\mathcal{A}_{\sigma}(p)\operatorname{diag}(G,I_{k})$ for any $k$-element subset $\sigma\subset[n]$ which implies that any $k$-focal of $\mathcal{A}G$ differs from the same $k$-focal of $\mathcal{A}$ by a factor of $\det(G)\neq 0$. ∎

From the proof of Lemma 2.3, we see that a $\mathbb{P}^{3}$ coordinate change that sends $\mathbf{q}\mapsto G\mathbf{q}$ maps $k$-focals to $k$-focals, picking up only a scalar factor $\det G\neq 0$. We will now see that change of coordinates on the image planes $\mathbb{P}^{2}$ affect the $k$-focals in a more subtle way.

Let $\mathcal{G}=(G_{1},\ldots,G_{n})\in(GL_{3})^{n}$ be a sequence of invertible matrices and consider the camera arrangement $\mathcal{G}\mathcal{A}:=(G_{1}A_{1},\ldots,G_{n}A_{n})$ obtained from a given arrangement $\mathcal{A}$ by left-multiplying $A_{i}$ with $G_{i}$. Note that the focal point of the camera $A_{i}$ is the same as the focal point of the camera $G_{i}A_{i}$. Since $p_{i}=(x_{i},y_{i},z_{i})$, we denote the ring $\mathbb{C}[x_{1},y_{1},z_{1},\ldots,x_{i},y_{i},z_{i},\ldots,x_{n},y_{n},z_{n}]$ by $\mathbb{C}[p_{1},\ldots,p_{n}]$ and a polynomial in it by $f(p_{1},\ldots,p_{n})$. The sequence $\mathcal{G}$ induces a camera-wise linear change of coordinates $\chi_{\mathcal{G}}$ on $\mathbb{C}[p_{1},\ldots,p_{n}]$ by sending

[TABLE]

Note that this amounts to a change of coordinates in the image planes $\mathbb{P}^{2}$ of the cameras in $\mathcal{A}$. Let $G^{-1}p$ denote $\chi_{\mathcal{G}}(p)=(G_{1}^{-1}p_{1},\ldots,G_{n}^{-1}p_{n})$. In what follows we will also need the notation $\mathcal{G}^{-1}:=(G_{1}^{-1},\ldots,G_{n}^{-1})$, $\mathcal{G}^{-1}\mathcal{A}:=(G_{1}^{-1}A_{1},\ldots,G_{n}^{-1}A_{n})$ and $\chi_{\mathcal{G}^{-1}}(p_{i})=G_{i}p_{i}$.

To analyze the effect of $\chi_{\mathcal{G}}$ on $k$-focal ideals, we recall the classical Cauchy-Binet formula, a proof of which can be found in [4].

Lemma 2.4** (Cauchy-Binet).**

If $A$ and $B$ are rectangular matrices of size $m\times n$ and $n\times m$, respectively, where $m\leq n$, then the determinant of the square matrix $AB$ is:

[TABLE]

where $:$ indicates that all rows/columns are taken.

Lemma 2.5**.**

For the $k$-focal ideal $H_{\mathcal{A}}^{k}$, $\chi_{\mathcal{G}}(H_{\mathcal{A}}^{k})=H_{\mathcal{G}\mathcal{A}}^{k}$. Similarly, $\chi_{\mathcal{G}^{-1}}(H_{\mathcal{G}\mathcal{A}}^{k})=H_{\mathcal{A}}^{k}$.

Proof.

We prove the first statement and the other follows similarly. We will show that the $k$-focal ideal of $\mathcal{A}_{[k]}$ is sent to the $k$-focal ideal of $(\mathcal{GA})_{[k]}$. The result then follows for the full $k$-focal ideal $H_{\mathcal{A}}^{k}$ by summing the $k$-focal ideals of all $\mathcal{A}_{\sigma}$ as $\sigma$ varies over all $k$-subsets of $[n]$.

Recall that a $k$-focal polynomial of $\mathcal{A}_{[k]}:=(A_{1},\dots,A_{k})$ is a maximal minor of:

[TABLE]

Applying $\chi_{\mathcal{G}}$ to this maximal minor is the same as taking the same maximal minor of

[TABLE]

The corresponding $k$-focal polynomial of $\mathcal{G}\mathcal{A}$ is the same maximal minor of

[TABLE]

The ideal $\chi_{\mathcal{G}}(H_{\mathcal{A}_{[k]}}^{k})$ is generated by the maximal minors of $\mathcal{A}_{[k]}(\chi_{\mathcal{G}}(p))$, namely

[TABLE]

while $H^{k}_{(\mathcal{G}\mathcal{A})_{[k]}}$ is generated by the maximal minors of $(\mathcal{G}\mathcal{A})_{[k]}(p)$. We need to show that these ideals coincide.

Let $G$ denote the block diagonal matrix with blocks $G_{1},\ldots,G_{n}$. A $(4+k)$-minor of $(\mathcal{G}\mathcal{A})_{[k]}(p)$ is the determinant of a submatrix with $4+k$ rows indexed by some $\tau\in{[3k]\choose 4+k}$. Such a submatrix has the form $G_{\tau}\mathcal{A}_{[k]}(G^{-1}p)$ where $G_{\tau}$ is the submatrix of $G$ consisting of the rows of $G$ indexed by $\tau$. By the Cauchy-Binet formula,

[TABLE]

This implies that $\det(G_{\tau}\mathcal{A}_{[k]}(G^{-1}p))$ lies in the ideal $\chi_{\mathcal{G}}(H_{\mathcal{A}_{[k]}}^{k})$, and hence, $H^{k}_{(\mathcal{G}\mathcal{A})_{[k]}}\subseteq\chi_{\mathcal{G}}(H_{\mathcal{A}_{[k]}}^{k})$.

The reverse containment follows by applying the same argument to $\mathcal{A}_{[k]}(p)=G^{-1}G\mathcal{A}_{[k]}(p)$ and $G\mathcal{A}_{[k]}(p)$ where $G^{-1}$ is the block diagonal matrix with blocks $G_{1}^{-1},\ldots,G_{k}^{-1}$.

Summing over all $k$ camera subsets, the result follows:

[TABLE]

∎

This proof shows that, contrary to popular belief, it is not true that $k$-focal polynomials go to $k$-focal polynomials under the change of coordinates given by $\chi_{\mathcal{G}}$, but the ideals do as in Lemma 2.5.

3. The Multiview Ideal

Recall from Definition 1.2 that the multiview ideal $M_{\mathcal{A}}$ of the camera arrangement $\mathcal{A}$ is the vanishing ideal of $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$, meaning that it is the set of all polynomials in $\mathbb{C}[p_{1},\ldots,p_{n}]$ that vanish on $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$. Since $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$ is a subset of $(\mathbb{P}^{2}_{\mathbb{R}})^{n}$, $M_{\mathcal{A}}$ is, in fact, generated by polynomials with real coefficients111Let $h(x)=f(x)+ig(x)$ be a complex polynomial, where $f(x)$ and $g(x)$ are real polynomials. Then if $h(x)$ vanish on a set of real points, then so must $f(x)$ and $g(x)$..

The complex projective variety $\mathbf{V}(M_{\mathcal{A}})\subset(\mathbb{P}^{2})^{n}$, which is the complex Zariski closure of $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$, is the multiview variety of $\mathcal{A}$. One might wonder if it is better to study the real Zariski closure of $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$ and its vanishing ideal since complex points in the multiview variety do not have any physical meaning, and hence no relevance to multiview geometry. However, observe that if the real Zariski closure was strictly smaller than the set of real points in $\mathbf{V}(M_{\mathcal{A}})$, then there would be a polynomial not in $M_{\mathcal{A}}$ that vanishes on $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$, which would contradict that $M_{\mathcal{A}}$ is the vanishing ideal of $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$. Therefore, $M_{\mathcal{A}}$ is also the vanishing ideal of the real Zariski closure of $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$, and hence a real radical ideal [18, §12.5].

Further, since $\varphi_{\mathcal{A}}$ is a polynomial map and $\mathbb{P}_{\mathbb{R}}^{3}$ is irreducible, $\mathbf{V}(M_{\mathcal{A}})$ is an irreducible three-dimensional variety in $(\mathbb{P}^{2})^{n}$. Hence $M_{\mathcal{A}}$ is a prime (homogeneous) ideal, meaning that if $fg\in M_{\mathcal{A}}$ then either $f$ or $g$ is in $M_{\mathcal{A}}$.

It was shown in [2] that the bifocals, trifocals and quadrifocals of $\mathcal{A}$ form a universal Gröbner basis of $M_{\mathcal{A}}$ under a certain genericity assumption on the cameras. This means that this collection of polynomials form a Gröbner basis for $M_{\mathcal{A}}$ with respect to any term order [6]. We will use this result to establish a generating set for $M_{\mathcal{A}}$ when the camera foci are distinct.

We first note what happens to $M_{\mathcal{A}}$ under the change of coordinates $\chi_{\mathcal{G}}$ defined in the previous section. Recall that $\chi_{\mathcal{G}}$ sends a polynomial $f(p_{1},\ldots,p_{n})\in\mathbb{C}[p_{1},\ldots,p_{n}]$ to $f(G_{1}^{-1}p_{1},\ldots,G_{n}^{-1}p_{n})$.

Lemma 3.1**.**

The image of the multiview ideal $M_{\mathcal{A}}$ under the map $\chi_{\mathcal{G}}$ is $M_{\mathcal{G}\mathcal{A}}$, the multiview ideal of $\mathcal{G}\mathcal{A}$. i.e. , $\chi_{\mathcal{G}}(M_{\mathcal{A}})=M_{\mathcal{G}\mathcal{A}}$. Similarly, $\chi_{\mathcal{G}^{-1}}(M_{\mathcal{G}\mathcal{A}})=M_{\mathcal{A}}$.

Proof.

Again, we will prove that $\chi_{\mathcal{G}}(M_{\mathcal{A}})=M_{\mathcal{G}\mathcal{A}}$. The proof that $\chi_{\mathcal{G}^{-1}}(M_{\mathcal{G}\mathcal{A}})=M_{\mathcal{A}}$ is similar.

From the definition we see that a polynomial $f(p_{1},\ldots,p_{n})$ vanishes on the multiview variety $\mathbf{V}(M_{\mathcal{A}})$ if and only if $f(A_{1}\mathbf{q},\ldots,A_{n}\mathbf{q})=0$ for all $\mathbf{q}\in\mathbb{P}^{3}\smallsetminus\{\mathbf{c}_{1},\ldots,\mathbf{c}_{n}\}$, equivalently, if and only if

[TABLE]

for all $\mathbf{q}\in\mathbb{P}^{3}\smallsetminus\{\mathbf{c}_{1},\ldots,\mathbf{c}_{n}\}$. The multiview variety of $\mathcal{G}\mathcal{A}$ is the Zariski closure of the points $(G_{1}A_{1}\mathbf{q},\ldots,G_{n}A_{n}\mathbf{q})$ as $\mathbf{q}$ varies over $\mathbb{P}^{3}\smallsetminus\{\mathbf{c}_{1},\ldots,\mathbf{c}_{n}\}$. Therefore, $f$ vanishes on $\mathbf{V}(M_{\mathcal{A}})$ if and only if $\chi_{\mathcal{G}}(f)$ vanishes on $\mathbf{V}(M_{\mathcal{G}\mathcal{A}})$. This proves that $\chi_{\mathcal{G}}(M_{\mathcal{A}})\subseteq M_{\mathcal{G}\mathcal{A}}$.

To finish the proof we need to argue that if $g(p_{1},\ldots,p_{n})\in M_{\mathcal{G}\mathcal{A}}$ then $g=\chi_{\mathcal{G}}(f)$ for some $f\in M_{\mathcal{A}}$. A polynomial $g\in M_{\mathcal{G}\mathcal{A}}$ if and only if $g(G_{1}A_{1}\mathbf{q},\ldots,G_{n}A_{n}\mathbf{q})=0$ for all $\mathbf{q}\in\mathbb{P}^{3}\smallsetminus\{\mathbf{c}_{1},\ldots,\mathbf{c}_{n}\}$ if and only if $g(G_{1}\mathbf{p}_{1},\ldots,G_{n}\mathbf{p}_{n})=0$ for all $(\mathbf{p}_{1},\ldots,\mathbf{p}_{n})\in\mathbf{V}(M_{\mathcal{A}})$. Define $g(G_{1}p_{1},\ldots,G_{n}p_{n})=:f\in M_{\mathcal{A}}$. Then $\chi_{\mathcal{G}}(f)=g(p_{1},\ldots,p_{n})$. ∎

We will use the results obtained so far to give an elementary proof that the bifocals and trifocals generate the multiview ideal $M_{\mathcal{A}}$ for any arrangement $\mathcal{A}$ of cameras with pairwise distinct foci. An important tool will be translational cameras.

Definition 3.2**.**

A camera $T$ is said to be translational if its left $3\times 3$ block is the identity matrix, i.e. , $T=[I\,\,\mathbf{t}]$ for some $\mathbf{t}\in\mathbb{R}^{3}$.

Lemma 3.3**.**

If $\mathcal{T}$ is an arrangement of translational cameras, then $H^{4}_{\mathcal{T}}\subseteq H^{3}_{\mathcal{T}}$.

Proof.

Using Macaulay2, this statement can be checked for $n=4$ translational cameras with foci represented symbolically as $(t_{i1},t_{i2},t_{i3},-1)$. For $n\geq 4$, since $H^{4}_{\mathcal{T}}=\sum_{\sigma\in{[n]\choose 4}}H^{4}_{\mathcal{T}_{\sigma}}$ and $H^{3}_{\mathcal{T}}=\sum_{\sigma\in{[n]\choose 3}}H^{3}_{\mathcal{T}_{\sigma}}$, the statement follows. ∎

We now use translational cameras to show that the quadrifocals are not needed in a generating set of $M_{\mathcal{A}}$. This is done by extending the result for translational cameras to finite cameras. Recall that a finite camera is a camera whose left $3\times 3$ block is invertible, or equivalently a camera whose focal point is not a point at infinity. Observe that any finite camera can be obtained by multiplying some translational camera on the left by an invertible $3\times 3$ matrix.

Corollary 3.4**.**

If $\mathcal{A}$ is any arrangement of cameras, then $H^{4}_{\mathcal{A}}\subseteq H^{3}_{\mathcal{A}}$.

Proof.

If $\mathcal{A}$ is an arrangement of finite cameras, then $A_{i}=G_{i}[I\,\,\mathbf{t}_{i}]$ for some $G_{i}\in GL_{3}$. Therefore $\mathcal{A}=\mathcal{G}\mathcal{T}$ where $\mathcal{T}$ is an arrangement of translational cameras. By Lemma 3.3, $H^{4}_{\mathcal{T}}\subseteq H^{3}_{\mathcal{T}}$. Hence, Lemma 2.5 implies

[TABLE]

For any four cameras indexed by $\sigma\in{[n]\choose 4}$, there exists some $G\in\operatorname{GL}_{4}$ which takes the foci of $\mathcal{A}_{\sigma}$ off of the plane at infinity, i.e. , so that $\mathcal{A}_{\sigma}G$ is an arrangement of finite cameras. Inverting this $\mathbb{P}^{3}$-coordinate change does not change ideal containment by Lemma 2.3. The general result follows since $H_{\mathcal{A}}^{4}=\sum_{\sigma\in{[n]\choose 4}}H_{\mathcal{A}_{\sigma}}^{4}\subseteq\sum_{\sigma\in{[n]\choose 3}}H_{\mathcal{A}_{\sigma}}^{3}=H^{3}_{\mathcal{A}}.$ ∎

To get to our main result, we will need a result from [2] about camera arrangements $\mathcal{A}$ that are generic in the sense that all $4\times 4$ minors of $[A_{1}^{\top}\,A_{2}^{\top}\,\cdots\,A_{n}^{\top}]$ are non-zero. We call such an $\mathcal{A}$ minor-generic.

Corollary 3.5**.**

Suppose $\mathcal{A}$ is minor-generic. Then $M_{\mathcal{A}}=H_{\mathcal{A}}^{2}+H_{\mathcal{A}}^{3}$.

Proof.

Theorem 2.1 in [2] says that if $\mathcal{A}$ is minor-generic, then the bifocals, trifocals and quadrifocals form a universal Gröbner basis of $M_{\mathcal{A}}$. In particular, this implies that $M_{\mathcal{A}}=H_{\mathcal{A}}^{2}+H_{\mathcal{A}}^{3}+H_{\mathcal{A}}^{4}$. The statement is then immediate from Corollary 3.4. ∎

Minor-genericity is a purely algebraic condition on camera arrangements. The following statement, which appears as a brief comment in [2] without proof, gives a geometric reinterpretation of this condition.

Lemma 3.6**.**

If $\mathcal{A}$ is minor-generic, then the foci of the cameras in $\mathcal{A}$ are pairwise distinct. Conversely, if the cameras in $\mathcal{A}$ have pairwise distinct foci, then there exist $G_{i}\in\textup{GL}_{3}$ such that $\mathcal{G}\mathcal{A}$ is minor-generic.

Proof.

Let $L_{i}\subset\mathbb{C}^{4}$ denote the three-dimensional row span of $A_{i}$. If $A_{i}$ and $A_{j}$ have the same focal point then $L_{i}=L_{j}$ and hence any four of the six rows of $A_{i}$ and $A_{j}$ are linearly dependent and $\mathcal{A}$ is not minor-generic. This proves the first statement.

Now suppose the foci of cameras in $\mathcal{A}$ are pairwise distinct. This means that the planes $L_{i}$ are pairwise distinct. For any $G_{i}\in\textup{GL}_{3}$, the rows of $G_{i}A_{i}$ form a basis of $L_{i}$. By choosing $G_{i}$ appropriately, the three rows of $A_{i}$ can be sent to any choice of three linearly independent vectors in $L_{i}$. We need to show that there is a choice of $G_{i}$ such that no four rows from the matrices $G_{i}A_{i}$ are linearly dependent.

Consider the $3n\times 4$ matrix obtained by vertically stacking the cameras in $\mathcal{A}$, as a point in $(\mathbb{C}^{4})^{3n}$, with coordinates $x_{kl}^{i}$ representing the $(k,l)$-entry of the $i$th camera. We will identify this point in $(\mathbb{C}^{4})^{3n}$ with the corresponding $3n\times 4$ matrix, and stack of $n$ cameras, and call all of them $\mathcal{A}$. Let $\mathcal{A}(x)$ denote the symbolic $3n\times 4$ matrix with entries $x^{i}_{kl}$. For $\sigma\in{[3n]\choose 4}$, let $d_{\sigma}$ denote the determinant of the $4\times 4$ submatrix of $\mathcal{A}(x)$ with rows indexed by $\sigma$. These cut out ${3n\choose 4}$ quartic hypersurfaces $\mathbf{V}(d_{\sigma})$ in $(\mathbb{C}^{4})^{3n}$. Let $v_{i}$ denote the normal of the hyperplane $L_{i}\subset\mathbb{C}^{4}$. Impose linear conditions saying that the rows of $\mathcal{A}(x)$, numbered $3i,3i+1,3i+2$, dot to zero with $v_{i}$. These $3n$ equations determine a subspace $L$ in $(\mathbb{C}^{4})^{3n}$ of dimension at least $9n=12n-3n$. The given point $\mathcal{A}$ lies in $L$. We need to show that there is a choice of $\mathcal{G}\in(\textup{GL}_{3})^{n}$ such that $\mathcal{G}\mathcal{A}$ (which again lies in $L$) avoids the determinantal surfaces. This is equivalent to picking a basis for each $L_{i}$ that stack together to a $\mathcal{B}\in L\smallsetminus\bigcup_{\sigma}\mathbf{V}(d_{\sigma})$.

We first show that $L$ is not contained in any $\mathbf{V}(d_{\sigma})$ by exhibiting a point in $L\smallsetminus\mathbf{V}(d_{\sigma})$ for each $\sigma$. Since at most four cameras can be involved in any $d_{\sigma}$, we may assume without loss of generality that $\sigma$ involves only rows of the first four cameras. There are four cases to consider depending on how many rows these four cameras contribute to $\sigma$ — the possibilities being $(3,1,0,0)$, $(2,2,0,0)$, $(2,1,1,0)$, and $(1,1,1,1)$. In each case we will produce a $\mathcal{B}\in L\smallsetminus\mathbf{V}(d_{\sigma})$. A key observation is that $A_{i}$ and $A_{j}$ having distinct foci implies $L_{i}\cap L_{j}$ is a proper subspace of both $L_{i}$ and $L_{j}$ for all $i,j$. Our starting point in each case below is $\mathcal{A}\in L$ which we modify to the needed $\mathcal{B}$ by replacing the bases of $L_{i}$ that provide the rows of $A_{i}$.

Case 1.     (3,1,0,0): Modify $\mathcal{A}$ to $\mathcal{B}$ by choosing a basis for $L_{2}$ to be the three rows of $B_{2}$ so that no element in this basis lies in $L_{1}\cap L_{2}$. Then $\mathcal{B}$ does not vanish on $d_{\sigma}$.

Case 2.     (2,2,0,0): Choose a basis for $L_{1}$ such that the two rows $v_{1},v_{2}$ contributing to $\sigma$ from the first camera are chosen from $L_{1}\setminus L_{2}$. Then $L_{2}\cap\operatorname{Span}\{v_{1},v_{2}\}$ is a proper subspace of $L_{2}$ of dimension at most one. Therefore taking two linearly independent vectors $v_{3},v_{4}$ outside of this subspace as the two rows from $L_{2}$ creates a $\mathcal{B}$ that does not vanish on $d_{\sigma}$.

Case 3.    (2,1,1,0): Choose a basis for $L_{1}$ such that the two contributing rows $v_{1},v_{2}$ from the first camera lie in $L_{1}\setminus(L_{2}\cup L_{3})$. Choose the row $v_{3}$ from $L_{2}$ such that $v_{3}\in L_{2}\setminus(\operatorname{Span}\{v_{1},v_{2}\}\cup L_{3})$, which forces $L_{3}\cap\operatorname{Span}\{v_{1},v_{2},v_{3}\}$ to be a proper subspace of $L_{3}$. Taking $v_{4}$ outside this subspace, we get a point $\mathcal{B}\in L$ at which $d_{\sigma}$ does not vanish.

Case 4.    (1,1,1,1): Choose $v_{1}\in L_{1}\smallsetminus(L_{2}\cup L_{3}\cup L_{4})$, $v_{2}\in L_{2}\smallsetminus(\operatorname{Span}\{v_{1}\}\cup L_{3}\cup L_{4})$, $v_{3}\in L_{3}\smallsetminus(\operatorname{Span}\{v_{1},v_{2}\}\cup L_{4})$, and $v_{4}\in L_{4}\smallsetminus(\operatorname{Span}\{v_{1},v_{2},v_{3}\})$. By construction, we get a point in $L$ at which $d_{\sigma}$ does not vanish.

Therefore, $L\cap\mathbf{V}(d_{\sigma})$ is a proper subvariety of $L$ for each $\sigma$, and a generic choice of $\mathcal{G}$ will put $\mathcal{G}\mathcal{A}\in L\smallsetminus\bigcup_{\sigma}\mathbf{V}(d_{\sigma})$. ∎

We note that $\mathcal{A}$ having distinct foci does not imply that $\mathcal{A}$ is minor-generic. A simple example would be an arrangement of four translational cameras; the submatrix consisting of the four first rows in each camera has zero determinant. However, having distinct foci allows the camera arrangement to be made minor-generic by the action of a tuple $\mathcal{G}$. We are now ready to prove the main theorem of this section.

Theorem 3.7**.**

Let $\mathcal{A}$ be an arrangement of cameras with distinct foci. Then $M_{\mathcal{A}}=H^{2}_{\mathcal{A}}+H^{3}_{\mathcal{A}}$.

Proof.

By Lemma 3.6, there exists $\mathcal{G}\in(GL_{3})^{n}$ such that $\mathcal{G}\mathcal{A}$ is minor-generic. Then, by Corollary 3.5, $M_{\mathcal{G}\mathcal{A}}=H^{2}_{\mathcal{G}\mathcal{A}}+H^{3}_{\mathcal{G}\mathcal{A}}$. Therefore, by Lemmas 3.1 and 2.5, we get

[TABLE]

∎

Proposition 5(1) in [21] says that the $H^{2}_{\mathcal{A}}$ and $H^{3}_{\mathcal{A}}$ together cut out the multiview variety which implies that $H^{2}_{\mathcal{A}}+H^{3}_{\mathcal{A}}\subseteq\mathcal{M}_{\mathcal{A}}$. Theorem 3.7 shows that these polynomials also generate the multiview ideal providing the analogous ideal-theoretic statement.

Theorem 3.7 improves on Corollary 2.7 in [2] which states that when the foci of the cameras $A_{i}$ are in linearly general position, then $M_{\mathcal{A}}$ is generated by the bifocals and trifocals. Theorem 3.7 requires no sophisticated condition on the cameras beyond the foci being pairwise distinct.

Conca et al.  [5] and Li [15] also consider the vanishing ideal of the image of linear map from a projective space to a product of projective spaces. It is shown in [5] that this ideal is Cartwright-Sturmfels, meaning that its initial ideal is radical after a generic change of coordinates. Both of these works allow for projective spaces of arbitrary dimension. Specializing to our situation, Li’s results show that $M_{\mathcal{A}}=\sum_{k=2}^{n}H_{\mathcal{A}}^{k}$ while we prove that $M_{\mathcal{A}}=H_{\mathcal{A}}^{2}+H_{\mathcal{A}}^{3}$.

Just like in [21] where the results automatically generalized from projective cameras to Euclidean cameras, Theorem 3.7 also generalizes to Euclidean cameras. Recall that a camera $A_{i}$ is Euclidean if it is of the form $A_{i}=[R_{i}\,\,t_{i}]$ where $R_{i}\in\textup{SO}_{3}$.

Corollary 3.8**.**

Let $\mathcal{A}$ be an arrangement of Euclidean cameras with pairwise distinct foci. Then $M_{\mathcal{A}}=H^{2}_{\mathcal{A}}+H^{3}_{\mathcal{A}}$.

We state one more consequence of Theorem 3.7 which will be needed in the next section.

Corollary 3.9**.**

Let $\mathcal{A}$ be a camera arrangement with pairwise distinct foci. Then for any $\mathbf{p}_{i}\in\mathbb{P}^{2}$, the points $(A_{1}\mathbf{c}_{i},A_{2}\mathbf{c}_{i},\dots,\mathbf{p}_{i},\dots,A_{n}\mathbf{c}_{i})$ lie in $\mathbf{V}(M_{\mathcal{A}})$ where $\mathbf{c}_{i}$ is the focal point of $A_{i}$.

Proof.

By Theorem 3.7, it suffices to show that for any $i$, the bifocals and trifocals vanish on the points $(A_{1}\mathbf{c}_{i},A_{2}\mathbf{c}_{i},\dots,\mathbf{p}_{i},\dots,A_{n}\mathbf{c}_{i})$. For any pair of cameras $\{i,j\}$, observe that $(\mathbf{c}_{i},0,-1)$ is a nonzero element of $\ker\mathcal{A}_{\{i,j\}}(\mathbf{p}_{i},A_{j}\mathbf{c}_{i})$. For any pair $\{j,k\}$ not containing camera $i$, $(\mathbf{c}_{i},-1,-1)$ is a nonzero element of $\ker\mathcal{A}_{\{j,k\}}(A_{j}\mathbf{c}_{i},A_{k}\mathbf{c}_{i})$. Hence all polynomials of $H_{\mathcal{A}}^{2}$ vanish on $(A_{1}\mathbf{c}_{i},A_{2}\mathbf{c}_{i},\dots,\mathbf{p}_{i},\dots,A_{n}\mathbf{c}_{i})$. A similar argument applies to any triples of cameras, from which it follows that all polynomials in $H_{\mathcal{A}}^{3}$ vanish on $(A_{1}\mathbf{c}_{i},A_{2}\mathbf{c}_{i},\dots,\mathbf{p}_{i},\dots,A_{n}\mathbf{c}_{i})$. ∎

The image of focal point $i$ in image $j$, i.e. , $A_{j}\mathbf{c}_{i}$, is called the epipole in image $j$ relative to image $i$. Corollary 3.9 shows that while the product of an arbitrary point in image $i$ with all epipoles relative to image $i$ does not appear in the image of $\varphi_{\mathcal{A}}$, these points appear in the multiview variety after taking Zariski closure. See also Proposition 1 in [21].

We conclude this section by showing that the hypothesis in Theorem 3.7 cannot be relaxed, namely if a pair of foci of cameras in $\mathcal{A}$ coincide, then the multiview ideal is strictly larger than the ideal generated by bifocals and trifocals.

Example 3.10**.**

Consider the four translational camera arrangement $\mathcal{A}$ where $\mathbf{t}_{1},\mathbf{t}_{2}=(0,0,0)$, $\mathbf{t}_{3}=(1,1,1)$, $\mathbf{t}_{4}=(-1,-1,-1)$. Eliminating the variables $q$ and $\lambda_{i}$ from the ideal $\langle A_{i}q-\lambda_{i}p_{i}:i=1,\dots,n\rangle$, we can directly obtain $M_{\mathcal{A}}$. Computing a primary decomposition of $H^{2}_{\mathcal{A}}+H^{3}_{\mathcal{A}}$, we find that

[TABLE]

The extra component $\langle{y}_{4}-{z}_{4},{y}_{3}-{z}_{3},{x}_{4}-{z}_{4},{x}_{3}-{z}_{3}\rangle$ cuts out the points $(\mathbf{p}_{1},\mathbf{p}_{2},A_{3}\mathbf{c}_{1},A_{4}\mathbf{c}_{1})$, and from the primary decomposition we see that the projective variety they form is not contained in $\mathbf{V}(M_{\mathcal{A}})$.

4. More Ideals for the Multiview Variety

In the computer vision literature, there are several sets of polynomials that have been shown to vanish on the space of images $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})$, and hence they also vanish on the multiview variety. We now consider three such sets of polynomials and the ideals they generate, and compare them to the multiview ideal $M_{\mathcal{A}}$.

4.1. Heyden and Åström [12]

Heyden and Åström were the first to do an algebraic study of the multiview variety, by studying the $n$-focal ideal $H^{n}_{\mathcal{A}}$ [12]. The variety of this ideal is indeed the multiview variety.

Lemma 4.1**.**

For any camera arrangement $\mathcal{A}$ with pairwise distinct foci, $\mathbf{V}(M_{\mathcal{A}})=\mathbf{V}(H_{\mathcal{A}}^{n})$.

Proof.

Recall from the image formation equations, $A_{i}\mathbf{q}=\lambda_{i}\mathbf{p}_{i}$ for all $i=1,\ldots,n$, that if $\mathbf{p}=(\mathbf{p}_{1},\ldots,\mathbf{p}_{n})$ lies in the image of $\varphi_{\mathcal{A}}$ then the matrix $\mathcal{A}(\mathbf{p})$ has a non-trivial kernel. This means that all maximal minors of $\mathcal{A}(p)$ vanish on the image of $\varphi_{\mathcal{A}}$, and therefore also on its Zariski closure, which is the multiview variety. Therefore, $\mathbf{V}(M_{\mathcal{A}})\subseteq\mathbf{V}(H_{\mathcal{A}}^{n})$.

To see the reverse inclusion, suppose $\mathbf{p}=(\mathbf{p}_{1},\ldots,\mathbf{p}_{n})\in\mathbf{V}(H_{\mathcal{A}}^{n})$ which means that $\mathcal{A}(\mathbf{p})$ is rank deficient and there is a nonzero vector of the form $(\mathbf{q},-\lambda_{1},\ldots,-\lambda_{n})$ in the kernel of $\mathcal{A}(\mathbf{p})$. If $\mathbf{q}=0$, then we will get that $\lambda_{i}\mathbf{p}_{i}=0$ for all $i$. However, since $\mathbf{p}_{i}\neq 0$, it must be that $\lambda_{i}=0$ for all $i$ and hence the vector in the kernel is the zero vector which is a contradiction. Therefore, there is a nonzero vector $\mathbf{q}$ such that $A_{i}\mathbf{q}=\lambda_{i}\mathbf{p}_{i}$ for some $\lambda_{i}$. If $\mathbf{q}$ is not the focal point of any camera, then $\mathbf{p}$ lies in $\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{C}}^{3})$. Since $\varphi_{\mathcal{A}}$ is continuous, $\varphi_{\mathcal{A}}(\overline{\mathbb{P}_{\mathbb{R}}^{3}})\subseteq\overline{\varphi_{\mathcal{A}}(\mathbb{P}_{\mathbb{R}}^{3})}$. It follows that $\varphi_{\mathcal{A}}(\mathbb{P}^{3}_{\mathbb{C}})\subseteq\mathbf{V}(M_{\mathcal{A}})$ because $\overline{\mathbb{P}_{\mathbb{R}}^{3}}=\mathbb{P}_{\mathbb{C}}^{3}$ and so $\mathbf{p}\in\mathbf{V}(M_{\mathcal{A}})$. On the other hand, if $\mathbf{q}$ is the focal point $\mathbf{c}_{i}$ of camera $i$, then $\mathbf{p}_{j}=A_{j}\mathbf{c}_{i}$ for all $j\neq i$, and by Corollary 3.9, $\mathbf{p}\in\mathbf{V}(M_{\mathcal{A}})$. Thus we get that $\mathbf{V}(M_{\mathcal{A}})\supseteq\mathbf{V}(H_{\mathcal{A}}^{n})$.

∎

Example 3.10 shows that the assumption of distinct foci is necessary for Lemma 4.1. In this example, $n=4$ and $H^{4}_{\mathcal{A}}=H^{2}_{\mathcal{A}}+H^{3}_{\mathcal{A}}$ by Corollary 3.4. We see that $\mathbf{V}(H^{4}_{\mathcal{A}})$ has a component other than $\mathbf{V}(M_{\mathcal{A}})$.

4.2. Faugeras et al. [8].

The second set of polynomials we will study were constructed by Faugeras & Mourrain while proving that the multiview variety is cut out by epipolar/bifocal and trifocal polynomials, and that the quadrifocal constraints corresponding to the quadrifocal tensor were not needed [7, 8].

Observe that $A_{i}\mathbf{q}=\lambda_{i}\mathbf{p}_{i}$ implies $A_{i}\mathbf{q}\times\mathbf{p}_{i}=0$, for each $i$, or equivalently, $[p_{i}]_{\times}A_{i}\mathbf{q}=0$, where

[TABLE]

represents taking cross product with $p_{i}$, i.e. , $[p_{i}]_{\times}v=p_{i}\times v$. Stacking all $3\times 4$ matrices $[p_{i}]_{\times}A_{i}$, we get the $3n\times 4$ partially symbolic matrix

[TABLE]

If there is a world point $\mathbf{q}$ satisfying $A_{i}\mathbf{q}\times\mathbf{p}_{i}=0$, then this matrix is rank deficient and all maximal minors of $\mathcal{A}_{F}(p)$ vanishes on the multiview variety.

Definition 4.2**.**

The ideal of all maximal $4\times 4$ minors of $\mathcal{A}^{F}(p)$, denoted by $F_{\mathcal{A}}$, will be called the Faugeras ideal of the arrangement $\mathcal{A}$. We denote the subideals of $F_{\mathcal{A}}$ generated by minors involving only two and three cameras by $F^{2}_{\mathcal{A}}$ and $F_{\mathcal{A}}^{3}$, respectively.

We now describe a sequence of matrix transformations that allow us to obtain $\mathcal{A}^{F}(p)$ from $\mathcal{A}(p)$. Let $P(p):=\operatorname{diag}([p_{1}]_{\times},\dots,[p_{n}]_{\times})$ be the symbolic block diagonal matrix of size $3n\times 3n$. Multiplying $\mathcal{A}(p)$ on the left by the block diagonal matrix $P(p)$ and dropping the rightmost $n$ columns of the resulting matrix, we obtain $\mathcal{A}^{F}(p)$:

[TABLE]

where as before, we abuse notation to let $\mathcal{A}$ also represents the $3n\times 4$ matrix $[A_{1};\dots;A_{n}]$ obtained by stacking the cameras vertically. From the matrix constructions of $H_{\mathcal{A}}^{n}$ and $F_{\mathcal{A}}$, we observe that their projective vanishing sets in $(\mathbb{P}^{2})^{n}$ coincide.

Lemma 4.3**.**

For any camera arrangement $\mathcal{A}$ with pairwise distinct foci, $\mathbf{V}(M_{\mathcal{A}})=\mathbf{V}(F_{\mathcal{A}})$.

Proof.

The proof will follow from Lemma 4.1 if we can show that $\mathbf{V}(F_{\mathcal{A}})=\mathbf{V}(H_{\mathcal{A}}^{n})$. If $\mathbf{p}\in(\mathbb{P}^{2})^{n}$ is such that $\mathcal{A}^{F}(\mathbf{p})$ drops rank, then there exists a nonzero $\mathbf{q}\in\ker(\mathcal{A}^{F}(\mathbf{p}))$ so that $A_{i}\mathbf{q}\times\mathbf{p}_{i}=0$ for all $i$. This means there exist nonzero scale factors $\lambda_{i}$ such that $A_{i}\mathbf{q}=\lambda_{i}\mathbf{p}_{i}$. The vector $(\mathbf{q},-\lambda_{1},\dots,-\lambda_{n})$ is a nontrivial element in $\ker(\mathcal{A}(\mathbf{p}))$, so $\mathcal{A}(\mathbf{p})$ is rank deficient. Therefore $\mathbf{V}(F_{\mathcal{A}})\subseteq\mathbf{V}(H^{n}_{\mathcal{A}})$

For the other inclusion, if there is a nontrivial $(\mathbf{q},-\lambda_{1},\dots,-\lambda_{n})\in\ker(\mathcal{A}(\mathbf{p}))$ for some $\mathbf{p}\in(\mathbb{P}^{2})^{n}$, then as in the proof of Lemma 4.1, $\mathbf{q}$ must be nonzero, and so $\mathbf{q}$ is a nontrivial element of $\ker(\mathcal{A}^{F}(\mathbf{p}))$. This shows that $\mathbf{V}(F_{\mathcal{A}})\supseteq\mathbf{V}(H^{n}_{\mathcal{A}})$, hence $\mathbf{V}(F_{\mathcal{A}})=\mathbf{V}(H_{\mathcal{A}}^{n})=\mathbf{V}(M_{\mathcal{A}})$.

∎

4.3. Ma et al. [17]

The third and final set of polynomials we will study are the so called multiview rank constraints which were proposed by Ma and collaborators [17] as an alternative to the multilinear constraints studied for example in Hartley & Zisserman [11].

Suppose $A_{1}=[I\,\,0]$ and $A_{i}=[B_{i}\,\,\mathbf{t}_{i}]$ for $i\geq 2$. Starting with $\mathcal{A}(p)$, a series of matrix operations are described in Chapter 8 in [17] to arrive at a new set of determinantal polynomials, arising as maximal minors of

[TABLE]

Definition 4.4**.**

The ideal of all maximal $2\times 2$ minors of $\mathcal{A}^{Y}(p)$, denoted by $Y_{\mathcal{A}}$, will be called the Ma ideal of the arrangement $\mathcal{A}$.

We observe that $\mathcal{A}^{Y}(p)$ can be obtained from $\mathcal{A}^{F}(p)$ by multiplying by a single matrix on the right:

[TABLE]

From this we observe that $Y_{\mathcal{A}}$ has the same projective vanishing set as $F_{\mathcal{A}}$, and hence $H_{\mathcal{A}}^{n}$ and $M_{\mathcal{A}}$.

Lemma 4.5**.**

For any camera arrangement $\mathcal{A}$ with pairwise distinct foci and $A_{1}=[I\,\,0]$, $\mathbf{V}(M_{\mathcal{A}})=\mathbf{V}(Y_{\mathcal{A}})$.

Proof.

If $\mathbf{p}\in(\mathbb{P}^{2})^{n}$ is such that $\mathcal{A}^{Y}(\mathbf{p})$ drops rank, then there exists a nontrivial $({v}_{1},{v}_{2})\in\ker(\mathcal{A}^{Y}(\mathbf{p}))$. Therefore, $\mathbf{q}=({v}_{1}\mathbf{p}_{1},{v}_{2})\in\ker(\mathcal{A}^{F}(\mathbf{p}))$ is nontrivial. Note that it is necessary that we assume $A_{1}=[I\,\,0]$ so that $[\mathbf{p}_{1}]_{\times}A_{1}(v_{1}\mathbf{p}_{1},v_{2})=v_{1}[\mathbf{p}_{1}]_{\times}\mathbf{p}_{1}=0$. This shows that $\mathbf{V}(Y_{\mathcal{A}})\subseteq\mathbf{V}(F_{\mathcal{A}})$.

For the other inclusion, if $0\neq\mathbf{q}\in\ker(\mathcal{A}^{F}(\mathbf{p}))$ for some $\mathbf{p}\in(\mathbb{P}^{2})^{n}$, then since $\mathbf{p}_{1}\times[I\,\,0]\mathbf{q}=0$, there exists a scalar $v_{1}$ such that $v_{1}\mathbf{p}_{1}=(\mathbf{q}_{1},\mathbf{q}_{2},\mathbf{q}_{3})$. This means that $(v_{1},\mathbf{q}_{4})\in\ker(\mathcal{A}^{Y}(\mathbf{p}))$, which is nontrivial because if $v_{1}=0$, then $(\mathbf{q}_{1},\mathbf{q}_{2},\mathbf{q}_{3})=0$, so $\mathbf{q}_{4}\neq 0$. This shows $\mathbf{V}(Y_{\mathcal{A}})\supseteq\mathbf{V}(F_{\mathcal{A}})$, and the desired result follows from Lemma 4.3.

∎

Observe that $Y_{\mathcal{A}}$ is generated by polynomials of total degree 3. This fact has an interesting consequence. As we mentioned earlier, $Y_{\mathcal{A}}$ has been proposed as an alternate algebraic foundation for multi-view geometry. From Lemma 4.5, we know that it cuts out the multiview variety. Since $M_{\mathcal{A}}$ is the vanishing ideal of the multiview variety, we get that $Y_{\mathcal{A}}\subseteq M_{\mathcal{A}}$. However, from Theorem 3.7 we know that $M_{\mathcal{A}}=H^{2}_{\mathcal{A}}+H^{3}_{\mathcal{A}}$, i.e. it is generated by polynomials of degree two and three, which means that in general $Y_{\mathcal{A}}\neq M_{\mathcal{A}}$ and instead $Y_{\mathcal{A}}\subset M_{\mathcal{A}}$ or equivalently $Y_{\mathcal{A}}\subset H^{2}_{\mathcal{A}}+H^{3}_{\mathcal{A}}$. This means that the bifocals and trifocals imply the multiview rank constraints, but not the other way around. Similarly, $H^{n}_{\mathcal{A}}$ and $F_{\mathcal{A}}$, which are generated by polynomials of total degree $n$ and four respectively, are properly contained in $M_{\mathcal{A}}$. We see this in Example 4.6 below.

4.4. Relationships to the Multiview Ideal

We now compute the three ideals on an example, foreshadowing their structural properties, which we examine next.

Example 4.6**.**

Consider the translational arrangement $\mathcal{A}$ where $\mathbf{t}_{1}=(0,0,0)$, $\mathbf{t}_{2}=(1,0,0)$, $\mathbf{t}_{3}=(0,1,0)$ whose multiview ideal is:

[TABLE]

The primary decompositions of $H^{n}_{\mathcal{A}}$, $F_{\mathcal{A}}$, and $Y_{\mathcal{A}}$ are

[TABLE]

[TABLE]

where $C$ is a component minimally generated by 133 polynomials of total degree up to eight.

While each of $H^{n}_{\mathcal{A}}$, $F_{\mathcal{A}}$, and $Y_{\mathcal{A}}$ notably contains $M_{\mathcal{A}}$ as a component, the nature of their other components is worth further investigation. ∎

To analyze the extra components, we rely on several notions from commutative algebra, which we define next. The first notion is that of a multigraded ring. Consider the ring $\mathbb{C}[p_{1},\dots,p_{n}]$ endowed with the $\mathbb{Z}^{n}$-grading $\deg(w_{i})=\mathbf{e}_{i}$ where $w_{i}\in\{x_{i},y_{i},z_{i}\}$ and $\mathbf{e}_{i}$ is the $i$th standard basis vector in $\mathbb{R}^{n}$. We say a polynomial in this ring is homogeneous if each of its terms have the same multidegree.

The irrelevant ideal in this grading, which we denote by $\mathfrak{m}$, is the intersection of the ideals $\mathfrak{m}_{i}:=\langle x_{i},y_{i},z_{i}\rangle$:

[TABLE]

Observe that $\mathfrak{m}$ is generated by all multilinear monomials of multidegree $(1,1,\ldots,1)$ and total degree $n$. It is the maximal ideal in the ring $\mathbb{C}[p_{1},\dots,p_{n}]$ generated by homogeneous elements of strictly positive multidegree.

The radical of an ideal $I$ is the ideal $\sqrt{I}:=\{f:f^{k}\in I\text{ for some }k\in\mathbb{N}\}$. If $I$ is a homogeneous ideal then so is its radical, and $I\subseteq\sqrt{I}$. The colon of an ideal $I$ with the ideal $J$, denoted as $(I:J)$ is the set of all polynomials $f$ such that $fg\in I$ for all $g\in J$, i.e. , $I:J=\{f\,:\,fJ\subseteq I\}.$

Recall that the projective varieties of the ideals $H^{n}_{\mathcal{A}}$, $F_{\mathcal{A}}$, and $Y_{\mathcal{A}}$ all agree and equal the multiview variety $\mathbf{V}(M_{\mathcal{A}})$. We can now state a first relationship among the ideals that follows easily from the projective Nullstellensatz in our multigraded setting, whose statement and proof will appear in Appendix A.

Theorem 4.7**.**

For any $\mathcal{A}$ with pairwise distinct foci,

1. a)

$\sqrt{H^{n}_{\mathcal{A}}}:\mathfrak{m}=M_{\mathcal{A}}$. 2. b)

$\sqrt{F_{\mathcal{A}}}:\mathfrak{m}=M_{\mathcal{A}}$. 3. c)

$\sqrt{Y_{\mathcal{A}}}:\mathfrak{m}=M_{\mathcal{A}}$* when $A_{1}=[I\,\,0]$.*

Proof.

See Appendix A. ∎

In the language of algebraic geometry what this says is that $\sqrt{H^{n}_{\mathcal{A}}},\sqrt{F_{\mathcal{A}}}$ and $\sqrt{Y_{\mathcal{A}}}$ all cut out the multiview variety scheme-theoretically. They are not equal as ideals but they agree in high enough multidegree with $M_{\mathcal{A}}$, see [10, pp 50].

We now strengthen Theorem 4.7 (a) and (b) to show that the operation of taking the radical is not needed, i.e. , $H^{n}_{\mathcal{A}}\,:\,\mathfrak{m}=M_{\mathcal{A}}$ and $F_{\mathcal{A}}\,:\,\mathfrak{m}=M_{\mathcal{A}}$. This means that $H^{n}_{\mathcal{A}}$ and $F_{\mathcal{A}}$ already cut out the multiview variety scheme-theoretically. Experimental evidence suggests that when $A_{1}=[I\>|\;0]$, such a result is also true for $Y_{\mathcal{A}}$, but an explicit proof is made difficult by the convoluted structure of the $2\times 2$ minors of $\mathcal{A}^{Y}(p)$.

We first show that the simple structure of the primary decomposition of $H^{n}_{\mathcal{A}}$ observed in Example 4.6 holds in general.

Lemma 4.8**.**

For any camera arrangement $\mathcal{A}$ with pairwise distinct foci, $H^{n}_{\mathcal{A}}=M_{\mathcal{A}}\cap\mathfrak{m}$. In particular, $H^{n}_{\mathcal{A}}$ is a radical ideal with prime decomposition $M_{\mathcal{A}}\cap\mathfrak{m}_{1}\cap\mathfrak{m}_{2}\cap\cdots\cap\mathfrak{m}_{n}$.

Proof.

Suppose $f$ is a generator of $\in H^{n}_{\mathcal{A}}$, i.e. , a maximal minor of $\mathcal{A}(p)$. Then $f\in\mathfrak{m}$. Also, since $f$ vanishes on $\mathbf{V}(M_{\mathcal{A}})$, $f\in M_{\mathcal{A}}$. Therefore, $H^{n}_{\mathcal{A}}\subseteq M_{\mathcal{A}}\cap\mathfrak{m}$.

Now suppose $f\in M_{\mathcal{A}}\cap\mathfrak{m}$. Since $M_{\mathcal{A}}$ is generated by bifocals and trifocals $f=\sum\lambda_{i}r_{i}b_{i}+\sum\mu_{j}s_{j}t_{j}$ where $b_{i}$’s are bifocals, $t_{j}$’s are trifocals, $r_{i},s_{j}$ are monomials, and $\lambda_{i},\mu_{j}$ are scalars. Further, since $f\in\mathfrak{m}$, every term in $f$ is divisible by some generator $\prod_{i=1}^{n}w_{i}$ of $\mathfrak{m}$ where $w_{i}\in\{x_{i},y_{i},z_{i}\}$. Now consider $r_{i}b_{i}$. Since $b_{i}$ involves only two cameras, it must be that $r_{i}$ contains a variable $w_{i}$ from each of the other $n-2$ cameras so that each term of $r_{i}b_{i}$ lies in $\mathfrak{m}$. This makes $r_{i}b_{i}$ a monomial multiple of a $n$-focal by Lemma 2.2. The same argument holds for $s_{j}t_{j}$. Thus, $f\in H^{n}_{\mathcal{A}}$. ∎

Proposition b3 in [22] proves that when $\mathcal{A}$ is minor-generic, $H^{n}_{\mathcal{A}}$ is a radical ideal. Lemma 4.8 shows that $H^{n}_{\mathcal{A}}$ is always a radical ideal under the weaker assumption of distinct foci.

Theorem 4.9**.**

For any camera arrangement $\mathcal{A}$ with pairwise distinct foci, $H^{n}_{\mathcal{A}}:\mathfrak{m}=M_{\mathcal{A}}$.

Proof.

We first note that $M_{\mathcal{A}}:\mathfrak{m}=M_{\mathcal{A}}$. Suppose $f\in M_{\mathcal{A}}:\mathfrak{m}$. Then $fu\in M_{\mathcal{A}}$ for any monomial generator $u$ of $\mathfrak{m}$. Since $M_{\mathcal{A}}$ is prime and does not contain any monomials, $f\in M_{\mathcal{A}}$. Since $H^{n}_{\mathcal{A}}=M_{\mathcal{A}}\cap\mathfrak{m}$ by Lemma 4.8, $H^{n}_{\mathcal{A}}:\mathfrak{m}=M_{\mathcal{A}}:\mathfrak{m}=M_{\mathcal{A}}$. ∎

We now consider the Faugeras ideal $F_{\mathcal{A}}$ and prove that $F_{\mathcal{A}}:\mathfrak{m}=M_{\mathcal{A}}$. The nontrivial part is to argue that $M_{\mathcal{A}}$ is contained in $F_{\mathcal{A}}:\mathfrak{m}$. This fact relies on the following technical lemma, similar in flavor to Lemma 2.2, which shows that bifocals and trifocals can both be multiplied by any generator of $\mathfrak{m}$ to fall into $F_{\mathcal{A}}$.

Lemma 4.10**.**

1. a)

For $n=2$ cameras, and any monomial $p_{1j}p_{2k}$, there exists a $4\times 4$ minor $f$ of $\mathcal{A}^{F}(p)$ such that $f=(-1)^{j+k}p_{1j}p_{2k}\det\mathcal{A}(p)$. 2. b)

Let $n=3$ and $i_{1},i_{2},i_{3}$ be pairwise distinct. Then for any trifocal $\det\mathcal{A}(p)_{\{p_{i_{1}j_{1}}p_{i_{2}j_{2}}\}}$ and any coordinate $p_{i_{3}k}$, there exists a $4\times 4$ minor $f$ of $\mathcal{A}^{F}(p)$ such that $f=(-1)^{k}p_{i_{3}k}\det\mathcal{A}(p)_{\{p_{i_{1}j_{1}}p_{i_{2}j_{2}}\}}$.

Proof.

See Appendix B both for the notation and the proof. ∎

Theorem 4.11**.**

For any camera arrangement $\mathcal{A}$ with pairwise distinct foci, $F_{\mathcal{A}}:\mathfrak{m}=M_{\mathcal{A}}$.

Proof.

The containment $F_{\mathcal{A}}:\mathfrak{m}\subseteq M_{\mathcal{A}}$ follows as in Theorem 4.9 because $F_{\mathcal{A}}\subseteq M_{\mathcal{A}}$ and hence, $F_{\mathcal{A}}:\mathfrak{m}\subseteq M_{\mathcal{A}}:\mathfrak{m}=M_{\mathcal{A}}$. The other containment will follow by showing $H^{2}_{\mathcal{A}},H^{3}_{\mathcal{A}}\subseteq F_{\mathcal{A}}:\mathfrak{m}$. For general camera arrangements with $n$ cameras, recall that $F^{2}_{\mathcal{A}}$ (resp. $F^{3}_{\mathcal{A}}$) is the ideal generated by all $4\times 4$ minors of $\mathcal{A}^{F}(p)$ that involve only two (resp. three) cameras. By Lemma 4.10(a), for any multilinear monomial $(\prod_{m=1}^{n}w_{m})$ and any bifocal $b_{ij}$, $(\prod w_{m})b_{ij}\in(f)$ for some Faugeras minor $f\in F^{2}_{\mathcal{A}}$, hence $H^{2}_{\mathcal{A}}\subseteq F_{\mathcal{A}}:\mathfrak{m}$. We address the trifocals in two cases. First consider the case when the two rows eliminated from $\mathcal{A}_{\{i,j,k\}}(p)$ to form a trifocal $t\in H^{3}_{\{i,j,k\}}$ come from the same camera, say without loss of generality, from camera $i$. In this case, $t=w_{i}b_{jk}$ for some $w_{i}$, and Lemma 4.10(a) again implies $t\in F_{\mathcal{A}}:\mathfrak{m}$. For the case when the two rows from $\mathcal{A}_{\{i,j,k\}}(p)$ to form $t\in H^{3}_{\{i,j,k\}}$ come from different cameras, Lemma 4.10(b) implies that, for any $(\prod w_{m})$, $(\prod w_{m})t\in(f)$ for some $f\in F^{3}_{\mathcal{A}}$. We conclude that $H^{3}_{\mathcal{A}}\subseteq F_{\mathcal{A}}:\mathfrak{m}$, as desired. ∎

5. The Bifocal Ideal

We saw in Theorem 3.7 that the bifocals and trifocals together generate the multiview ideal when the camera foci are pairwise distinct. In this section, we investigate how imposing further conditions on the cameras can lead to an even simpler description of the multiview ideal. Heyden and Åström [12] and Trager et al.  [21] show that when the camera foci are not all on a plane, the bifocals are necessary and sufficient to cut out the multiview variety. There has also been work to further reduce this description by considering the minimal number of bifocals needed ([12], [23]), though we will not address this question here. In this section, we focus on the ideal-theoretic relationship between the bifocal ideal $H^{2}_{\mathcal{A}}$ and the multiview ideal $M_{\mathcal{A}}$ when the camera foci are noncoplanar.

To motivate our investigation, we start with some examples. We say that a camera arrangement $\mathcal{A}$ is coplanar, noncoplanar or collinear if their foci have the corresponding property.

Example 5.1**.**

Consider the four noncoplanar translational camera arrangement $\mathcal{A}_{1}$ where $\mathbf{t}_{1}=(0,0,0)$, $\mathbf{t}_{2}=(1,0,0)$, $\mathbf{t}_{3}=(0,1,0)$, $\mathbf{t}_{4}=(0,0,1)$. Eliminating the variables $q$ and $\lambda_{i}$ from the ideal $\langle A_{i}q-\lambda_{i}p_{i}:i=1,\dots,n\rangle$, we observe $M_{\mathcal{A}_{1}}$ occurs as a component in $H^{2}_{\mathcal{A}_{1}}$

[TABLE]

Example 5.2**.**

Consider the four coplanar translational camera arrangement $\mathcal{A}_{2}$ where $\mathbf{t}_{1}=(1,0,0)$, $\mathbf{t}_{2}=(0,1,0)$, $\mathbf{t}_{3}=(0,0,1)$, $\mathbf{t}_{4}=(1/3,1/3,1/3)$. We observe that $H_{\mathcal{A}_{2}}^{2}=M_{\mathcal{A}_{2}}\cap C$ where

[TABLE]

In Example 5.1, each extra component of $H^{2}_{\mathcal{A}_{1}}$ contains an irrelevant ideal $\mathfrak{m}_{i}$ and hence does not contribute to $\mathbf{V}(H^{2}_{\mathcal{A}_{1}})$. Saturating the bifocal ideal $H^{2}_{\mathcal{A}_{1}}$ with respect to the full irrelevant ideal $\mathfrak{m}$ removes these components. We will prove that this is always true when camera foci are noncoplanar. We begin by proving a series of three lemmas.

Lemma 5.3**.**

Suppose $\mathcal{A}$ is an arrangement of $n\geq 4$ cameras with pairwise distinct foci. Then $\mathcal{A}$ is noncoplanar $\implies$ $H_{\mathcal{A}}^{n}\subseteq H_{\mathcal{A}}^{2}$.

Proof.

$\mathbf{n=4,5,6}$. If $\mathcal{A}$ is noncoplanar, then there is some subset of four cameras that is noncoplanar. Order the cameras in $\mathcal{A}$ so that these are the cameras $A_{1},\dots,A_{4}$. By a change of coordinates on $\mathbb{P}^{3}$, we can send the foci of the cameras $A_{1},\dots,A_{4}$ to the foci of the cameras in $\mathcal{A}_{1}$ from Example 5.1. Then, by Lemma 2.5, applying $\mathbb{P}^{2}$ coordinate changes using some $\mathcal{G}\in(GL_{3})^{n}$, we can assume that $\mathcal{A}$ is an arrangement of translational cameras. These transformations fix the first four cameras, and we think of the cameras $A_{i}$ for $i\geq 5$ as variable, represented symbolically by their translations, and the implication can confirmed by direct calculation in Macaulay2.

$\mathbf{n=7}$. In this case, the full computation is too expensive. To make the computation feasible, we split the proof into two cases, depending on whether the arrangement has five collinear cameras or not.

Case I: If a noncoplanar arrangement of seven cameras has at most four collinear cameras, then every four camera subset can be augmented with two additional cameras to get a noncoplanar arrangement of six cameras. Thus every 7-focal of such an arrangement, which looks like $w_{i}w_{j}w_{k}q$ for some quadrifocal $q$, has the form of a 6-focal from a noncoplanar arrangement, say $w_{i}w_{j}q$, multiplied by a coordinate $w_{k}$. The $n=6$ case shows that $w_{i}w_{j}q$ is generated by 2-focals, hence $w_{i}w_{j}w_{k}q$ is generated by 2-focals.

Case II: We now consider the case of noncoplanar seven camera arrangements in which five cameras are collinear. In this case, by a proper choice of camera ordering and $\mathbb{P}^{3}$ coordinate change, we can assume the translations of $A_{5},A_{6},A_{7}$ are of the form $\mathbf{t}_{5}=(\lambda_{5},0,0)^{\top},\mathbf{t}_{5}=(\lambda_{6},0,0)^{\top},\mathbf{t}_{5}=(\lambda_{7},0,0)^{\top}$ where the $\lambda_{i}$ are symbolic. This makes $A_{1},A_{2},A_{5},A_{6},A_{7}$ collinear. The choice to take the line that the cameras lie on to be the $x$ axis is arbitrary, but can be made without loss of generality. This arrangement is now described by few enough variables to enable a direct computation showing that $H_{\mathcal{A}}^{7}\subseteq H_{\mathcal{A}}^{2}$.

$\mathbf{n\geq 8}$. Now suppose $n\geq 8$ and $f$ is an $n$-focal of $\mathcal{A}$. Recall that $f$ involves all $n$ cameras but at most four cameras can contribute two rows to the matrix whose determinant is $f$. At one extreme, these four cameras maybe $A_{1},\ldots,A_{4}$ and at the other extreme they might be four cameras different from the first four, which we call $A_{5},\ldots,A_{8}$. Thus the $n$-focal $f\in H_{\mathcal{A}}^{n}$ is a monomial multiple of a 8-focal $g=mq$ of $\{A_{1},\dots,A_{4},A_{5},\dots,A_{8}\}$ where where $q$ is a quadrifocal and $m$ is a monomial.

If the four cameras contributing to $q$ involve $A_{1},\dots,A_{4}$, then $g$ is a multiple of a 7-focal from noncoplanar cameras. On the other hand, if $q\in H_{A_{5},\dots,A_{8}}^{4}$, then $q$ can be generated by the trifocals of $A_{5},\ldots,A_{8}$ by Lemma 3.3:

[TABLE]

In particular, this shows that $g$ can be generated from 7-focals, $mt_{i}$. These come from noncoplanar seven camera arrangements because $A_{1},\dots,A_{4}$ are noncoplanar. In either case, we know that such 7-focals can be generated by 2-focals, hence $g\in H_{\mathcal{A}}^{2}$. It follows that $f\in H_{\mathcal{A}}^{2}$, as desired. ∎

Lemma 5.4**.**

Suppose $\mathcal{A}$ is an arrangement of $n\geq 4$ cameras with pairwise distinct foci. Then $H_{\mathcal{A}}^{n}\subseteq H_{\mathcal{A}}^{2}\implies M_{\mathcal{A}}=H_{\mathcal{A}}^{2}:\mathfrak{m}$.

Proof.

If $f\in H^{2}_{\mathcal{A}}:\mathfrak{m}$, then $f(\prod z_{i})\in H^{2}_{\mathcal{A}}\subseteq M_{\mathcal{A}}$, vanishes on $\mathbf{V}(M_{\mathcal{A}})$. Since $M_{\mathcal{A}}$ is prime and does not contain any monomials, $f\in M_{\mathcal{A}}$. Therefore, $H^{2}_{\mathcal{A}}:\mathfrak{m}\subseteq M_{\mathcal{A}}$. For the other containment, by Theorem 3.7, it suffices to show that $H^{2}_{\mathcal{A}}$ and $H^{3}_{\mathcal{A}}$ are contained in $H^{2}_{\mathcal{A}}:\mathfrak{m}$. It is clear that $H^{2}_{\mathcal{A}}\subseteq H^{2}_{A}:\mathfrak{m}$. By Lemma 2.2, multiplying any $t\in H^{3}_{\mathcal{A}}$ by a generator $\prod w_{i}$ of $\mathfrak{m}$ yields a monomial multiple of an $n$-focal. By assumption, this $n$-focal lies in $H_{\mathcal{A}}^{2}$. Thus, $t\in H^{2}_{A}:\mathfrak{m}$ and $M_{\mathcal{A}}\subseteq H^{2}_{A}:\mathfrak{m}$. ∎

Lemma 5.5**.**

Suppose $\mathcal{A}$ is an arrangement of $n\geq 4$ cameras with pairwise distinct foci. Then $M_{\mathcal{A}}=H_{\mathcal{A}}^{2}:\mathfrak{m}\implies\mathcal{A}$ is noncoplanar.

Proof.

We prove the contrapositive, namely that if $\mathcal{A}$ is coplanar then $M_{\mathcal{A}}\neq H^{2}_{A}:\mathfrak{m}$. We will construct a point $\mathbf{p}\in\mathbf{V}(H^{2}_{\mathcal{A}}:\mathfrak{m})\smallsetminus\mathbf{V}(M_{\mathcal{A}})$, from which the result will follow.

Let $\mathbf{n}\in\mathbb{P}^{3}$ be the normal vector of a plane containing the foci of the cameras in $\mathcal{A}$. If the foci are not collinear then $\mathbf{n}$ is unique, otherwise we choose any plane containing the foci and its normal $\mathbf{n}$. Let $l_{i}\subseteq\mathbb{P}^{2}$ denote the image of the plane $\mathbf{n}^{\perp}$ in camera $i$, and let $\mathbf{e}_{i,j}$ denote the image of the focal point of camera $j$ in image $i$. Then $\mathbf{e}_{i,j}\in l_{i}$ since the focal point of camera $j$ lies in $\mathbf{n}^{\perp}$. Choose $\mathbf{p}_{1}\in l_{1}\smallsetminus\{\mathbf{e}_{1,2},\mathbf{e}_{1,3}\}$ and $\mathbf{p}_{2}\in l_{2}\smallsetminus\{\mathbf{e}_{2,1},\mathbf{e}_{2,3}\}$. Then there is a unique world point $\mathbf{q}$ on $\mathbf{n}^{\perp}$ whose images in cameras $1$ and $2$ are $\mathbf{p}_{1}$ and $\mathbf{p}_{2}$. Let $\widetilde{\mathbf{p}}_{3}\in l_{3}$ be the (unique) image of $\mathbf{q}$ in camera $3$. Then $\mathbf{p}_{1},\mathbf{p}_{2},\widetilde{\mathbf{p}}_{3}$ satisfy trifocal constraints. Choose $\mathbf{p}_{3}\in l_{3}\smallsetminus\{\widetilde{\mathbf{p}}_{3}\}$ and some $\mathbf{p}_{i}\in l_{i}$ for $i\geq 4$. By construction, $\mathbf{p}\notin\mathbf{V}(M_{\mathcal{A}})$. Since the cameras are coplanar, the epipolar plane given by $\mathbf{q}$ and any two cameras $i$ and $j$ is $\mathbf{n}^{\perp}$ for any pair $i,j$. By choosing $\mathbf{p}_{i}\in l_{i}$ for all $i$, we force every bifocal polynomial to vanish on $\mathbf{p}$. Therefore by construction, $\mathbf{p}\in\mathbf{V}(H_{\mathcal{A}}^{2})\smallsetminus\mathbf{V}(M_{\mathcal{A}})$, but since $\mathbf{V}(H^{2}_{\mathcal{A}})=\mathbf{V}(H^{2}_{\mathcal{A}}:\mathfrak{m})$, we conclude that $H^{2}_{\mathcal{A}}:\mathfrak{m}\neq M_{\mathcal{A}}$. ∎

Together, Lemmas 5.3, 5.4, 5.5 imply the following theorem.

Theorem 5.6**.**

Suppose $\mathcal{A}$ is an arrangement of $n\geq 4$ cameras with pairwise distinct foci. Then the following are equivalent.

1. (a)

$\mathcal{A}$* is noncoplanar.* 2. (b)

$H_{\mathcal{A}}^{n}\subseteq H_{\mathcal{A}}^{2}$. 3. (c)

$M_{\mathcal{A}}=H_{\mathcal{A}}^{2}:\mathfrak{m}$.

We now make some observations about Theorem 5.6.

Theorem 6.1 in [12] observes that $\mathbf{V}(H_{\mathcal{A}}^{2})=\mathbf{V}(M_{\mathcal{A}})$ for noncoplanar $\mathcal{A}$ while Proposition 5 (2) in [21] further shows that $\mathbf{V}(H_{\mathcal{A}}^{2})=\mathbf{V}(M_{\mathcal{A}})$ is equivalent to the foci of $\mathcal{A}$ being noncoplanar. Our Theorem 5.6 proves the analogous ideal statement, namely that noncoplanarity of foci is equivalent to $M_{\mathcal{A}}=H^{2}_{\mathcal{A}}\,:\,\mathfrak{m}$.

Example 5.2 shows how Theorem  5.6 fails when $\mathcal{A}$ is coplanar. The bifocal ideal $H^{2}_{\mathcal{A}_{2}}$ contains the component $\langle x_{1}+y_{1}+z_{1},x_{2}+y_{2}+z_{2},x_{3}+y_{3}+z_{3},x_{4}+y_{4}+z_{4}\rangle$, which cannot be removed by saturating with respect to $\mathfrak{m}$. Its variety cuts out the projections of the plane containing the foci of $\mathcal{A}_{2}$ in each camera image. This plane in $\mathbb{P}^{3}$ has normal vector $(1,1,1,-1)$. The following example shows that further degeneracy occurs when camera foci are collinear.

Example 5.7**.**

Consider the four collinear translational camera arrangement $\mathcal{A}_{3}$ where $\mathbf{t}_{1}=(0,0,0)$, $\mathbf{t}_{2}=(1,0,0)$, $\mathbf{t}_{3}=(2,0,0)$, $\mathbf{t}_{4}=(3,0,0)$. Here, $H_{\mathcal{A}_{3}}^{2}\subseteq M_{\mathcal{A}_{3}}$, but both ideals are prime, so $M_{\mathcal{A}_{3}}$ cannot occur as a component of $H_{\mathcal{A}_{3}}^{2}$. In addition, the dimension of $H_{\mathcal{A}_{3}}^{2}$ is one larger than that of $M_{\mathcal{A}_{3}}$. This is explained by the fact that there is an entire one-dimensional family of planes that contains the camera centers of $\mathcal{A}_{3}$.

As seen in the above examples and discussion, the relation between $H_{\mathcal{A}}^{2}$ and $M_{\mathcal{A}}$ can be complicated when camera centers are coplanar or collinear. Determining the exact relationship between ideals in these degenerate settings would be an interesting problem for the future.

In Theorem 5.6 we showed that when cameras are noncoplanar, the $n$-focal ideal becomes a subset of the $2$-focal ideal. We now give an example to show that this containment need not hold for $H^{k}_{\mathcal{A}}$ where $n>k>2$. The construction relies on having three of five cameras being collinear.

Example 5.8**.**

Consider the five translational camera arrangement $\mathcal{B}$ with $\mathbf{t}_{1}=(0,0,0),\mathbf{t}_{2}=(0,0,1),\mathbf{t}_{3}=(0,0,2),\mathbf{t}_{4}=(0,1,0),\mathbf{t}_{5}=(0,0,1)$. Theorem 5.6 shows that $H_{\mathcal{B}}^{5}\subseteq H_{\mathcal{B}}^{2}$ since $\mathcal{B}$ is noncoplanar. However the following trifocal from $B_{1},B_{2},B_{3}$,

[TABLE]

is not in $H_{\mathcal{B}}^{2}$. Similarly, the quadrifocal,

[TABLE]

from cameras $B_{1},B_{2},B_{3},B_{4}$ is not in $H_{\mathcal{B}}^{2}$.

6. Finite Images

The results of the previous sections have important practical consequences when we restrict attention to the set of all finite images, that is to all $(\mathbf{p_{1}},\dots,\mathbf{p_{n}})\in\mathbf{V}(M_{\mathcal{A}})$ with $z_{i}\neq 0$ for all $i$. The vanishing ideal of this affine patch is obtained by dehomogenizing $M_{\mathcal{A}}$ with respect to the variables $z_{i}$ from each image plane. We call this the affine multiview ideal of $\mathcal{A}$ and denote it $\pi(M_{\mathcal{A}})$, where $\pi:\mathbb{C}[x_{i},y_{i},z_{i}]\to\mathbb{C}[x_{i},y_{i}]$ is the map setting each $z_{i}$ to 1. From Theorem 3.7, we see that $\pi(M_{\mathcal{A}})$ is generated by dehomogenized bifocals and dehomogenized trifocals when the foci of $\mathcal{A}$ are pairwise distinct.

Corollary 6.1**.**

If $\mathcal{A}$ is a camera arrangement with pairwise distinct foci, then $\pi(M_{\mathcal{A}})=\pi(H_{\mathcal{A}}^{2})+\pi(H_{\mathcal{A}}^{3})$.

Using the following fact about dehomogenizing colon ideals, the results of Section 4 yield a nice relation among $\pi(H_{\mathcal{A}}^{n}),\pi(F_{\mathcal{A}}),\pi(Y_{\mathcal{A}})$, and the affine multiview ideal, $\pi(M_{\mathcal{A}})$.

Lemma 6.2**.**

For ideals $I,J\subset\mathbb{C}[x_{i},y_{i},z_{i}]$, $\pi(I:J)=\pi(I):\pi(J)$.

Proof.

If $f\in\pi(I:J)$, then $f=\pi(g)$ for some $g$ which satisfies $gh\in I$ for all $h\in J$. Therefore $f\pi(h)=\pi(g)\pi(h)=\pi(gh)\in\pi(I)$ for any $h\in J$, proving $f\in\pi(I):\pi(J)$. If $f\in\pi(I):\pi(J)$, then for any $h\in J$, $f\pi(h)\in\pi(I)$, i.e. , there exists $g\in I$ such that $f\pi(h)=\pi(g)$. Denote the homogenization of $f$ with respect to $z_{1},\dots,z_{n}$ by $\widetilde{f}$. We claim that $\widetilde{f}\in I:J$. Indeed for any $h\in J$, $\pi(\widetilde{f}h)=\pi(\widetilde{f})\pi(h)=f\pi(h)=\pi(g)$ for some $g\in I$. Homogenizing both sides, we get $\widetilde{f}h=g\in I$, and we conclude that $\pi(I):\pi(J)\subseteq\pi(I:J)$ ∎

Corollary 6.3**.**

If $\mathcal{A}$ is a camera arrangement with pairwise distinct foci, then $\pi(M_{\mathcal{A}})=\pi(H_{\mathcal{A}}^{n})=\pi(F_{\mathcal{A}})=\pi(\sqrt{Y_{\mathcal{A}}})$.

Proof.

Lemma 6.2 implies that $\pi(I:\mathfrak{m})=\pi(I):(1)=\pi(I)$ for any ideal $I$. Dehomogenizing Theorems 4.9, 4.11, and 4.7, each equality follows. ∎

Observe that the last equality in Corollary 6.3 requires $A_{1}=[I\,\,0]$. Geometrically, Corollary 6.3 shows that while the homogenous ideals $H_{\mathcal{A}}^{n},F_{\mathcal{A}},Y_{\mathcal{A}}$, and $M_{\mathcal{A}}$ do not coincide, they are the same away from the origin in each image plane. In particular, this is the case on the affine patch $\{\mathbf{p}\in\mathbb{P}^{2n}:z_{1}=\dots=z_{n}=1\}$ corresponding to finite image data.

Using Theorem 5.6 we see that, when $\mathcal{A}$ is noncoplanar, the dehomogenized bifocals alone suffice to generate the affine multiview ideal $\pi(M_{\mathcal{A}})$.

Corollary 6.4**.**

Suppose $\mathcal{A}$ is a noncoplanar camera arrangement with pairwise distinct foci. Then

[TABLE]

Proof.

Dehomogenizing the result of Theorem 5.6, we get $\pi(M_{\mathcal{A}})=\pi(H_{\mathcal{A}}^{2}:\mathfrak{m})=\pi(H_{\mathcal{A}}^{2}):\pi(\mathfrak{m})=\pi(H_{\mathcal{A}}^{2}).$ ∎

Corollary 6.4 shows that $\pi(M_{\mathcal{A}})$ is generated by quadratics whenever $\mathcal{A}$ satisfies the noncoplanarity assumption. This observation was used in [1] to create a semidefinite programming relaxation of the triangulation problem which is can be seen as minimizing Euclidean distance from an observed noisy data point to the affine multiview variety. It was shown that when the noise is small, the semidefinite relaxation solves triangulation. Of course, Corollary 6.3 needs the foci of the cameras to be noncoplanar and indeed, the experiments in [1] show that the quality of the semidefinite programming solution deteriorates as the foci become coplanar and then collinear.

Geometrically, we can understand how the quality of the relaxation deteriorates because the bifocal ideal cuts out more than the multiview variety for coplanar arrangements. In the coplanar case, the bifocal ideal cuts out the image of the plane that contains the camera centers. These points are not the images of true 3D points. It is therefore possible that the nearest point problem yields a spurious solution on this extra component. Similarly, in the collinear case, the bifocal ideal cuts out a strictly larger variety than just the multiview variety. In this case, the dimension of the vanishing set of the bifocal ideal is one larger than the multiview variety.

7. Summary

The multiview variety is a foundational geometric object in multiview geometry and understanding its vanishing ideal $M_{\mathcal{A}}$ precisely is important for any algebraic algorithm that solves problems on this variety. There have been many partial results about the algebraic structure of the multiview variety. The aim of our paper is to put them all into a unified algebraic setting and give a complete description of $M_{\mathcal{A}}$.

Our main result is that when the foci of the cameras are pairwise distinct, $M_{\mathcal{A}}$ is generated by the bifocal and trifocal polynomials of $\mathcal{A}$ (Theorem 3.7). The proof requires an understanding of the behavior of coordinate changes on $k$-focal ideals (Lemma 2.5), and translational cameras (Lemma 3.3). The main result holds for Euclidean cameras as well (Corollary 3.8). We also give an example to illustrate that the assumption of distinct foci cannot be relaxed for this result to hold (Example 3.10).

Next we study three sets of polynomials that have been proposed to cut out the multiview variety, by Heyden-Åström, Faugeras and Ma et. al. respectively. We show that the ideals generated by these polynomials are all properly contained in $M_{\mathcal{A}}$. We establish the exact algebraic relationships between the above ideals and $M_{\mathcal{A}}$ (Theorems 4.7, 4.9 and 4.11).

We then prove that if the camera foci are assumed to be noncoplanar, then in fact $M_{\mathcal{A}}$ is the saturation of the bifocal ideal by the irrelevant ideal (Theorem 5.6). In this situation the $n$-focal ideal is a subset of the bifocal ideal.

Finally we prove that the dehomogenization of the ideals by Heyden-Åström, Faugeras and Ma et. al. all agree with the dehomogenization of $M_{\mathcal{A}}$ (Corollary 6.3). Similarly, under noncoplanarity of foci, the bifocal ideal also has the same dehomogenization (Corollary 6.4). This means that all of these ideals cut out the space of finite images.

8. Acknowledgements

We wish to thank the referees of this paper for their careful reading and suggestions. In particular, their comments helped fill a gap in the proof of the main theorem of Section 5.

Andrew Pryhuber and Rekha R. Thomas acknowledge support from the U.S. National Science Foundation through the grant DMS-1719538.

Appendix A: Multigraded Projective Nullstellensatz

In this appendix, we state and prove the projective Nullstellensatz in our multigraded setting, which we use to prove Theorem 4.7 in Section 4. Let $I\subseteq\mathbb{C}[p_{1},\dots,p_{n}]$ be homogeneous with respect to the $\mathbb{Z}^{n}$-grading $\deg w_{i}=\mathbf{e}_{i}$. To be clear about projective versus affine varieties, we define $\mathbf{V}_{\mathbb{P}}(I):=\mathbf{V}(I)=\{\mathbf{p}\in(\mathbb{P}^{2})^{n}:f(\mathbf{p})=0\text{ for all }f\in I\}$, and for a set $S\subseteq(\mathbb{P}^{2})^{n}$, we define

[TABLE]

We say that $\mathbf{V}_{\mathbb{P}}(I)$ is the projective vanishing set of $I$ in $(\mathbb{P}^{2})^{n}$ and $\mathbf{I}_{\mathbb{P}}(S)$ is the largest homogeneous ideal vanishing on $S$ contained in $\mathfrak{m}$. While we force $\mathbf{I}_{\mathbb{P}}(S)\subseteq\mathfrak{m}$, it also makes sense to consider the largest homogeneous ideal vanishing on $S$ without intersecting with $\mathfrak{m}$. As before we denote this ideal by $\mathbf{I}(S)$, and notice that $\mathbf{I}_{\mathbb{P}}(S)=\mathbf{I}(S)\cap\mathfrak{m}$. In the usual grading on $\mathbb{C}[p_{1},\ldots,p_{n}]$, a vanishing ideal $\mathbf{I}(S)$ is homogeneous in the usual sense which means that it is contained in the usual irrelevant ideal $\langle x_{1},y_{1},z_{1},\ldots,x_{n},y_{n},z_{n}\rangle$. Under the multi-grading, $\mathbf{I}_{\mathbb{P}}(S)$ is required to be in the corresponding irrelevant ideal $\mathfrak{m}$. We will use the following variant of the Nullstellensatz.

Lemma 8.1**.**

For any homogeneous ideal $I\subseteq\mathbb{C}[p_{1},\dots,p_{n}]$ such that $I\subseteq\mathfrak{m}$, $\mathbf{I}_{\mathbb{P}}(\mathbf{V}_{\mathbb{P}}(I))=\sqrt{I}.$

Proof.

Define the affine operations

[TABLE]

where we treat $S$ as a subset of $(\mathbb{A}^{3})^{n}$. We will use the affine version of the Nullstellensatz on the cone over $V:=\mathbf{V}_{\mathbb{P}}(I)$, i.e. , the set $C_{V}=\mathbf{V}_{\mathbb{A}}(I)\subseteq(\mathbb{A}^{3})^{n}$. We claim that

[TABLE]

First suppose $f\in\mathbf{I}_{\mathbb{A}}(C_{V})$. Given $\mathbf{p}=(\mathbf{p}_{1},\dots,\mathbf{p}_{n})\in V$, all homogeneous coordinates of $\mathbf{p}$, represented by scalings $(\lambda_{1}\mathbf{p}_{1},\dots,\lambda_{n}\mathbf{p}_{n})$, lie in $C_{V}$, so $f$ vanishes for all homogeneous coordinates of $\mathbf{p}$. This means that the homogeneous components $f_{i_{1},\dots,i_{n}}$ of $f$, consisting of all terms with multidegree $(i_{1},\dots,i_{n})$, vanish at $\mathbf{p}$, so $f\in\mathbf{I}(V)$, hence $\mathbf{I}_{\mathbb{A}}(C_{V})\subseteq\mathbf{I}(V)$. By the Nullstellensatz in $(\mathbb{A}^{3})^{n}$, $\mathbf{I}_{\mathbb{A}}(C_{V})=\mathbf{I}_{\mathbb{A}}(\mathbf{V}_{\mathbb{A}}(I))=\sqrt{I}$, and by the assumption that $I\subseteq\mathfrak{m}$, $\sqrt{I}\subseteq\sqrt{\mathfrak{m}}=\mathfrak{m}$. This shows that $\mathbf{I}_{\mathbb{A}}(C_{V})\subseteq\mathbf{I}(V)\cap\mathfrak{m}=\mathbf{I}_{\mathbb{P}}(V)$.

Conversely, suppose $f\in\mathbf{I}_{\mathbb{P}}(V)$. Since any point $\mathbf{p}$ of $C_{V}$ such that $\mathbf{p}_{i}\neq 0$ for all $i$ gives homogeneous coordinates for a point in $V$, it follows that $f$ vanishes on $C_{V}\smallsetminus\bigcup_{i=1}^{n}\mathbb{A}^{3}\times\dots\times\{0\}_{i}\times\dots\times\mathbb{A}^{3}$. We need to show that $f$ vanishes on each of the sets $\mathbb{A}^{3}\times\dots\times\{0\}_{i}\times\dots\times\mathbb{A}^{3}$. Since $f\subseteq\mathfrak{m}$, it has strictly positive multidegree, and every monomial in $f$ contains at least one coordinate from each copy of $\mathbb{A}^{3}$. Setting all 3 coordinates to zero in any $\mathbb{A}^{3}$ forces $f$ to be zero, so we conclude that $f\in\mathbf{I}_{\mathbb{A}}(C_{V})$. Finally, from (11), we conclude

[TABLE]

∎

Corollary 8.2**.**

For any homogeneous ideal $I\subseteq\mathbb{C}[p_{1},\dots,p_{n}]$, $\mathbf{I}_{\mathbb{P}}(\mathbf{V}_{\mathbb{P}}(I))=\sqrt{I}\cap\mathfrak{m}$.

Proof.

Observe that

[TABLE]

and

[TABLE]

Therefore by Lemma 8.1, $\mathbf{I}_{\mathbb{P}}(\mathbf{V}_{\mathbb{P}}(I))=\sqrt{I}\cap\mathfrak{m}$. ∎

Corollary 8.3**.**

For any $\mathcal{A}$ with pairwise distinct foci,

[TABLE]

Proof.

We have already shown in Section 4 that $\mathbf{V}_{\mathbb{P}}(H^{n}_{\mathcal{A}})=\mathbf{V}_{\mathbb{P}}(F_{\mathcal{A}})=\mathbf{V}_{\mathbb{P}}(Y_{\mathcal{A}})=\mathbf{V}_{\mathbb{P}}(M_{\mathcal{A}})$. Since $M_{\mathcal{A}}$ is radical, the result follows by Corollary 8.2. ∎

We can now prove Theorem 4.7, restated here, from the main body of the paper.

Theorem 8.4**.**

For any $\mathcal{A}$ with pairwise distinct foci,

1. a)

$\sqrt{H^{n}_{\mathcal{A}}}:\mathfrak{m}=M_{\mathcal{A}}$** 2. b)

$\sqrt{F_{\mathcal{A}}}:\mathfrak{m}=M_{\mathcal{A}}$** 3. c)

$\sqrt{Y_{\mathcal{A}}}:\mathfrak{m}=M_{\mathcal{A}}$* when $A_{1}=[I\;|\;0]$*

Proof.

Taking colon ideal with $\mathfrak{m}$, the desired result follows from Corollary 8.3 and the fact that $M_{\mathcal{A}}:\mathfrak{m}=M_{\mathcal{A}}$, which was proven in Theorem 4.9.

∎

Appendix B: Technical Proofs

In this appendix, we elaborate on the technical details used to prove Theorem 4.11. Recall that the nontrivial statement there was that bifocals and trifocals can be multiplied by any generator of $\mathfrak{m}$ to fall into $F_{\mathcal{A}}$. This requires understanding the $4\times 4$ minors of $\mathcal{A}^{F}(p)$ for which we once again invoke the Cauchy-Binet formula and the observation that $\mathcal{A}^{F}(p)=P(p)\mathcal{A}$ from (7).

First we characterize certain $4\times 4$ minors of $P(p)$. Let $p_{ij}$ denote the $j$th coordinate of $p_{i}$, i.e. , $p_{i1}=x_{i}$, $p_{i2}=y_{i}$, and $p_{i3}=z_{i}$. Having the subscript (resp. superscript) $p_{ij}$ on $P(p)$ indicates eliminating from $P(p)$ the unique row (resp. column) of $[p_{i}]_{\times}$ that does not contain $p_{ij}$. On the other hand, having the subscript $p_{ij}$ on the matrices $\mathcal{A}$ and $\mathcal{A}(p)$ will stand for eliminating the unique row of the matrix containing $p_{ij}$.

We will only need to consider the $4\times 4$ minors of $P(p)$ when $n=2$ and $n=3$. Let $R_{i},C_{i}\subseteq\{p_{i1},p_{i2},p_{i3}\}$ denote collections of coordinates, and write $R=\bigcup_{i=1}^{n}R_{i}$, $C=\bigcup_{i=1}^{n}C_{i}$. When $n=2$, a $4\times 4$ minor of $P(p)$ is $\det(P(p)_{R}^{C})$ for some $R$, $C$ of size $|R|=|C|=2$, and when $n=3$, $|R|=|C|=5$. Observe that if $|R_{i}|\neq|C_{i}|$ for any $i$, then the submatrix $P(p)_{R}^{C}$ has at least two linearly dependent rows or columns, yielding a zero minor. When $|R_{i}|=|C_{i}|$ for all $i$, $P(p)_{R}^{C}$ is block diagonal, so $\det(P(p)_{R}^{C})=\prod_{i=1}^{n}\det(([p_{i}]_{\times})_{R_{i}}^{C_{i}})$.

Lemma 8.5**.**

Let $n=2$. The nonzero $4\times 4$ minors of $P(p)$ are determined by collections of coordinates $R,C$ with $|R_{1}|=|C_{1}|=|R_{2}|=|C_{2}|=1$. For $R=\{p_{1j},p_{2k}\}$ and $C=\{p_{1l},p_{2m}\}$, the $4\times 4$ minor $\det(P(p)_{R}^{C})$ is the monomial

[TABLE]

Proof.

As noted above, if $|R_{i}|\neq|C_{i}|$ for either $i$, then $\det(P(p)_{R}^{C})=0$, whereas if $|R_{i}|=|C_{i}|=2$ for either $i$, then $P(p)_{R}^{C}$ has a rank 2 block on its diagonal, hence $\det(P(p)_{R}^{C})=0$, proving the first statement. For $R=\{p_{1j},p_{2k}\}$ and $C=\{p_{1l},p_{2m}\}$, the $4\times 4$ minor $\det P(p)_{R}^{C}$ is

[TABLE]

∎

Lemma 8.6**.**

Let $n=3$. Suppose $|R_{3}|=|C_{3}|=1$, and $|R_{1}|=|C_{1}|=|R_{2}|=|C_{2}|=2$. For $R_{3}=\{p_{3j}\},C_{3}=\{p_{3k}\}$, the $4\times 4$ minor $\det(P(p)_{R}^{C})$ is the monomial

[TABLE]

where $p_{1l}$ is the coordinate common to $R_{1}$ and $C_{1}$ and $p_{2m}$ is the coordinate common to $R_{2}$ and $C_{2}$.

Proof.

When $R_{i}=C_{i}$ as sets for $i=1$ or $i=2$, then $([p_{i}]_{\times})_{R_{i}}^{C_{i}}=0$, hence $\det P(p)_{R}^{C}=\prod_{i=1}^{n}\det(([p_{i}]_{\times})_{R_{i}}^{C_{i}})=0$. On the other hand, when $R_{1}\neq C_{1}$, $\det(([p_{1}]_{\times})_{R_{1}}^{C_{1}})=(-1)^{l}p_{1l}$ where $p_{1l}=R_{1}\cap C_{1}$. Similarly $\det(([p_{2}]_{\times})_{R_{2}}^{C_{2}})=(-1)^{m}p_{2m}$ where $p_{2m}=R_{2}\cap C_{2}$ when $R_{2}\neq C_{2}$. ∎

We now show that bifocals and trifocals can both be multiplied by any generator of $\mathfrak{m}$ to fall into $F_{\mathcal{A}}$.

Lemma 8.7**.**

1. a)

For $n=2$ cameras, and any monomial $p_{1j}p_{2k}$, there exists a $4\times 4$ minor $f$ of $\mathcal{A}^{F}(p)$ such that $f=(-1)^{j+k}p_{1j}p_{2k}\det(\mathcal{A}(p))$. 2. b)

Let $n=3$ and $i_{1},i_{2},i_{3}$ be pairwise distinct. Then for any trifocal $\det(\mathcal{A}(p)_{\{p_{i_{1}j_{1}}p_{i_{2}j_{2}}\}})$ and any coordinate $p_{i_{3}k}$, there exists a $4\times 4$ minor $f$ of $\mathcal{A}^{F}(p)$ such that $f=(-1)^{k}p_{i_{3}k}\det(\mathcal{A}(p)_{\{p_{i_{1}j_{1}}p_{i_{2}j_{2}}\}})$.

Proof.

(a) Fix some $p_{1j}p_{2k}$. Since $n=2$, $P(p)\mathcal{A}$ is a $6\times 4$ matrix and we need to delete two rows to get a $4\times 4$ minor. Using Lemma 8.5 and Cauchy-Binet, the result follows from the computation below:

[TABLE]

where the last equality follows from expanding the determinant of $\mathcal{A}(p)$ along the last two columns.

(b) Without loss of generality, let $i_{1}=1$, $i_{2}=2$, $i_{3}=3$ and let $p_{3k}$ be arbitrary. For simplicity, suppose $j_{1}=j_{2}=1$. Therefore, we consider the trifocal $\det(\mathcal{A}(p)_{\{p_{11},p_{21}\}})$. Using Lemma 8.6 and Cauchy-Binet, we expand $f=\det(P(p)_{R}\mathcal{A})$ where $R_{1}=\{p_{12},p_{13}\}$, $R_{2}=\{p_{22},p_{23}\}$, $R_{3}=\{p_{3k}\}$ as follows:

[TABLE]

[TABLE]

Observe that the final equality follows from expanding the determinant of $\mathcal{A}(p)_{\{p_{11},p_{21}\}}$ on the $p_{3}$ column.

For general $j_{1},j_{2}$, performing the same computation with $R_{1}=\{p_{11},p_{12},p_{13}\}\smallsetminus\{p_{1j_{1}}\}$, $R_{2}=\{p_{21},p_{22},p_{23}\}\smallsetminus\{p_{1j_{2}}\}$ and $R_{3}=\{p_{3k}\}$ yields $\det\left(P(p)_{R}\mathcal{A}\right)=(-1)^{k}p_{3k}\det\left(\mathcal{A}(p)_{\{p_{1j_{1}},p_{2j_{2}}\}}\right)$. ∎