Deterministic Completion of Rectangular Matrices Using Asymmetric Ramanujan Graphs: Exact and Stable Recovery

TL;DR

This paper introduces a deterministic sampling approach for matrix completion using asymmetric Ramanujan graphs, providing conditions for exact and stable recovery, and exploring phase transition phenomena in low-rank matrix reconstruction.

Contribution

It presents a novel deterministic sampling method for matrix completion based on Ramanujan graphs, with theoretical guarantees and numerical insights into phase transitions.

Findings

Exact recovery when Ramanujan graph degree ≥ 3 times the rank

Identified phase transition in success probability with increasing rank

Numerical simulations support theoretical conditions

Abstract

In this paper we study the matrix completion problem: Suppose $X \in {\mathbb R}^{n_r \times n_c}$ is unknown except for a known upper bound $r$ on its rank. By measuring a small number $m \ll n_r n_c$ of elements of $X$, is it possible to recover $X$ exactly with noise-free measurements, or to construct a good approximation of $X$ with noisy measurements? Existing solutions to these problems involve sampling the elements uniformly and at random, and can guarantee exact recovery of the unknown matrix only with high probability. In this paper, we present a \textit{deterministic} sampling method for matrix completion. We achieve this by choosing the sampling set as the edge set of an asymmetric Ramanujan bigraph, and constrained nuclear norm minimization is the recovery method. Specifically, we derive sufficient conditions under which the unknown matrix is completed exactly with noise-free measurements, and is approximately completed with noisy measurements, which we call "stable" completion. The conditions derived here are only sufficient and more restrictive than random sampling. To study how close they are to being necessary, we conducted numerical simulations on randomly generated low rank matrices, using the LPS families of Ramanujan graphs. These simulations demonstrate two facts: (i) In order to achieve exact completion, it appears sufficient to choose the degree $d$ of the Ramanujan graph to be $\geq 3r$. (ii) There is a "phase transition," whereby the likelihood of success suddenly drops from 100\% to 0\% if the rank is increased by just one or two beyond a critical value. The phase transition phenomenon is well-known and well-studied in vector recovery using $\ell_1$-norm minimization. However, it is less studied in matrix completion and nuclear norm minimization, and not much understood.

Tables

Table 1. Table 1 : Degree vs. critical rank in high degree LPS Ramanujan graphs

$d$	$\overline{r}$	$\overline{r}/d$
198	62	0.3147
230	75	0.3275
294	102	0.3481

Equations

$$(E_{\Omega})_{ij}=\left\{\begin{array}[]{ll}1&\mbox{if }(i,j)\in\Omega,\\ 0&\mbox{if }(i,j)\not\in\Omega.\end{array}\right.$$

$$\hat{X}=\underset{Z\in {\mathbb{R}}^{n_r\times n_c}}{\mathrm{argmin}}\mathrm{rank}(Z)\text{mbox}s.t.E_{\Omega }\circ Z=E_{\Omega }\circ X.$$

$$\hat{X}:=\underset{Z\in {\mathbb{R}}^{n_r\times n_c}}{\mathrm{argmin}}\Vert Z{\Vert }_N\text{mbox}s.t.E_{\Omega }\circ Z=E_{\Omega }\circ X.$$

$$\hat{X}:=\underset{Z\in {\mathbb{R}}^{n_r\times n_c}}{\mathrm{argmin}}\Vert Z{\Vert }_N\text{mbox}s.t.\Vert E_{\Omega }\circ Z-E_{\Omega }\circ X_{\mathcal{W}}{\Vert }_F\le ϵ$$

$${\mu }_0(U):=\frac{n_r}{r}\underset{i\in [n_r]}{\max }\Vert P_U{\mathbf{e}}_i{\Vert }_2^2=\frac{n_r}{r}\underset{i\in [n_r]}{\max }\Vert u^i{\Vert }_2^2,$$

$${\mu }_0(X):=\max \{{\mu }_0(U),{\mu }_0(V)\}.$$

$${\mu }_1(X):=\sqrt{\frac{n_r{n}_c}{r}}\Vert U{V}^{\top }{\Vert }_{\infty },$$

$$m\ge C{n}^{5/4}r\log (n)$$

$$\zeta =c{n}^{-3}$$

$$m\ge 32\max \{{\mu }_1^2,{\mu }_0\}r(n_r+n_c)\beta {\log }^2(2n_c)$$

$$\zeta =6\log (n_c)(n_r+n_c{)}^{2-2\beta }+n_c^{2-2\sqrt{\beta }},$$

$$m\ge C{\mu }_{0}rn{\log }^2(n_r+n_c)$$

$$\zeta =c_1(n_r+n_c{)}^{-c_2}$$

$$\zeta =c{n}^{-3}$$

$$\Vert \hat{X}-X{\Vert }_F\le \left[2\sqrt{\frac{C\min (n)}{p}}+1\right]2ϵ.$$

$$l\ge \max \left\{12\left(\log \frac{d}{ϵ}+1\right),2r\right\}.$$

$$\underset{G\in {\mathbb{R}}^{n_r\times r},H\in {\mathbb{R}}^{n_c\times r}}{\min }\Vert G{H}^{\top }-S{\Vert }_F^2+\lambda (\Vert G{\Vert }_F^2+\Vert H{\Vert }_F^2),$$

$$\underset{Z\in {\mathbb{R}}^{n_r\times n_c}}{\min }\Vert Z-S{\Vert }_F^2+\lambda \Vert Z{\Vert }_1,$$

$$L_{\sigma }(Z):=\frac{{\sigma }^2}{2}\sum _{(i,j)\in \Omega }[1-\exp (-(Z_{ij}-S_{ij}{)}^2/{\sigma }^2)],$$

$$y_l=a_l^{\top }X{a}_l+{\eta }_l=⟨X,a_l{a}_l^{\top }{⟩}_F+{\eta }_l,l\in [m],$$

$$X_{ij}=⟨X,{\mathbf{e}}_i{\mathbf{e}}_j^{\top }{⟩}_F,$$

$$y_l=⟨X,a_l{b}_l^{\top }{⟩}_F,l\in [m],$$

$$|{\lambda }_2|\le 2\sqrt{d-1}.$$

$$A=\left[\begin{array}[]{cc}0&B\\ B^{\top}&0\end{array}\right].$$

$$|{\sigma }_2|\le \sqrt{d_r-1}+\sqrt{d_c-1},$$

$${\Vert \sum _{k\in S}\frac{n_r}{d_c}(U^{k\top }{U}^k)-I_r\Vert }_S\le \theta ,\forall S\subseteq [n_r],|S|=d_c,$$

$${\Vert \sum _{k\in S}\frac{n_c}{d_r}(V^{k\top }{V}^k)-I_r\Vert }_S\le \theta ,\forall S\subseteq [n_c],|S|=d_r,$$

$$d\ge 36C^2{\mu }_0^2{r}^2,$$

$$d\ge 144{\mu }_0^2{r}^2.$$

$$\alpha =\frac{d_c}{n_r}=\frac{d_r}{n_c}=\sqrt{\frac{d_r{d}_c}{n_r{n}_c}},$$

Full text

The conditions derived here are only sufficient and more restrictive than random sampling. To study how close they are to being necessary, we conducted numerical simulations on randomly generated low rank matrices, using the LPS families of Ramanujan graphs. These simulations demonstrate two facts: (i) In order to achieve exact completion, it appears sufficient to choose the degree $d$ of the Ramanujan graph to be $\geq 3r$. (ii) There is a “phase transition,” whereby the likelihood of success suddenly drops from 100% to 0% if the rank is increased by just one or two beyond a critical value. The phase transition phenomenon is well-known and well-studied in vector recovery using $\ell_{1}$-norm minimization. However, it is less studied in matrix completion and nuclear norm minimization, and not much understood.

1 Introduction

1.1 General Statement

Compressed sensing refers to the recovery of high-dimensional but low-complexity objects from a small number of linear measurements. Recovery of sparse (or nearly sparse) vectors, and recovery of high-dimensional but low-rank matrices are the two most popular applications of compressed sensing. The object of study in the present paper is the matrix completion problem, which is a special case of low-rank matrix recovery. The matrix completion problem has been receiving attention due to its application to different areas such as rating systems or recommendation engines (also known as the “Netflix problem” [1], sensor localization, structure from motion [2], and quantum tomography [3, 4, 5, 6, 7, 8, 9]. In recommendation engines, the rows represent the products, ranging from the hundreds to the thousands, while the columns represent customers, ranging into the millions. In the community working on recommendation engines, it is believed, rightly or wrongly, that each customer uses no more than ten to fifteen features to make a selection; consequently, the matrix with all entries of customer preferences has rank no more than fifteen. In sensor localization, the idea is to infer the pairwise 3-D distance between $l$ different points, from measuring only a few pairwise distances. It can be shown that, no matter how large the number $l$ of points is, this matrix has rank no more than five. In quantum tomography, the dimensions can be huge, $2^{2^{n}}$ where $n$ is the number of qubits, but the matrix often has rank one or two. An excellent survey of the matrix completion problem can be found in [10].

1.2 Problem Definition

The matrix completion problem can be stated formally as follows: Suppose $X\in{\mathbb{R}}^{n_{r}\times n_{c}}$ is an unknown matrix that we wish to recover whose rank is bounded by a known integer $r$. Let $[n]$ denote the set $\{1,\ldots,n\}$ for each integer $n$. In the matrix completion problem, a set $\Omega\subseteq[n_{r}]\times[n_{c}]$ is specified, known as the sample set or measurement set. To be specific, suppose $\Omega=\{(i_{1},j_{1}),\ldots,(i_{m},j_{m})\}$, where $|\Omega|=m$ is the total number of samples. We are able to observe the values of the unknown matrix $X$ at the locations in the set $\Omega$, either with or without noise. In the noise-free case, the measurements consist of $X_{i,j}$ for all $(i,j)\in\Omega$. Equivalently, the measured matrix $S$ can be expressed as the Hadamard product111Recall that the Hadamard product $C$ of two matrices $A,B$ of equal dimensions is defined by $c_{ij}=a_{ij}b_{ij}$ for all $i,j$. $E_{\Omega}\circ X$ where $E_{\Omega}\in\{0,1\}^{n_{r}\times n_{c}}$ is defined by

[TABLE]

Note that if $X_{ij}$ happens to equal zero at some sampled location $(i,j)\in\Omega$, then the measured matrix $S:=E_{\Omega}\circ X$ will also equal zero at the same location. However, at the unsampled locations, $S=E_{\Omega}\circ X$ will always equal zero. In the exact matrix completion problem, the objective is to recover the unknown matrix $X$ exactly from $E_{\Omega}\circ X$. In the case of noisy measurements, the data available to the learner consists of $E_{\Omega}\circ X+{\cal W}$, where ${\cal W}$ is the measurement noise. In this case it is possible to make a distinction between two cases. In the robust matrix completion problem, it is assumed that the noise matrix ${\cal W}$ is sparse, that is, ${\cal W}_{ij}=0$ for all but a small number of pairs $(i,j)$. The objective is still to recover $X$ exactly. In the stable matrix completion problem, there are no assumptions on the number of nonzero elements of ${\cal W}$, but there is a known upper bound $\epsilon$ on the Frobenius norm of ${\cal W}$. In this case, exact recovery of $X$ may not be possible. Therefore, in stable matrix completion, the aim is to construct an accurate approximation to $X$.

In the case of noise-free measurements, one possible approach to the matrix completion problem is to set

[TABLE]

The above problem is a special case of minimizing the rank of an unknown matrix subject to linear constraints, and is therefore NP-hard [11]. Since the problem is NP-hard, a logical approach is to replace the rank function by its convex relaxation, which is the nuclear norm, or the sum of the singular values of a matrix, as shown in [12]. Therefore the convex relaxation of (1) is

[TABLE]

In the case of sparse noisy measurements, the above formulation can be still used (more in the section on literature review below). In the case of unstructured noisy measurements, let us define $X_{\cal W}=X+{\cal W}$, so that the noise-corrupted measurement consists of the Hadamard product $E_{\Omega}\circ X_{\cal W}$. In this case, as suggested in [13], (2) can be modified to

[TABLE]

where $\epsilon$ is a known upper bound on the Frobenius norm of the measurement error matrix ${\cal W}$. For instance, if ${\cal W}$ represents Gaussian noise with known characteristics, a bound on its Frobenius norm can be derived with a specified prior probability.

1.3 Contributions of the Present Paper

In the literature to date, most of the papers assume that the sample set $\Omega$ is chosen at random from $[n_{r}]\times[n_{c}]$, either without replacement as in [1, 14], or with replacement [15]. The authors are aware of only a handful of papers [16, 17, 18, 19, 20] in which a deterministic procedure is suggested for choosing the sample set $\Omega$. In [16, 17], the sample set $\Omega$ is chosen as the edge set of a Ramanujan graph. (This concept is defined below). Other references such as [19] suggest that the sample set $\Omega$ can be viewed as the edge set of a bipartite graph, but do not explicitly take advantage of this. In [17], the authors use nuclear norm minimization as in (2), and claim to derive some sufficient conditions. However, as shown in the Appendix, there is an error in that paper. In contrast, we present a correct sufficient condition for exact matrix completion. Moreover, our method of proof carries over seamlessly to stable matrix completion, which is not studied in [17]. In [16], the approach is max-norm minimization. The results there are “universal” in that they do not involve the coherence parameter of the unknown matrix $X$ (defined below), and also apply to the case where the elements are sampled with a nonuniform probability. The present authors have improved upon those bounds by a factor of two; see [21]. These improvements are achieved through modifying the so-called “expander mixing lemma” for bipartite graphs, which is independent interest. However, due to space limitations, the max norm minimization approach to exact matrix completion is not discussed in this paper.

2 Literature Review

There are two approaches to choosing the sample set $\Omega$, namely probabilistic and deterministic. In the probabilistic approach the elements of $\Omega$ are chosen at random from $[n_{r}]\times[n_{c}]$, usually, though not always, with a uniform distribution. In this setting one can further distinguish between two distinct situations, namely sampling from $[n_{r}]\times[n_{c}]$ with replacement or without replacement. In the deterministic setting, the sample set $\Omega$ is interpreted as the edge set of a bipartite graph.

2.1 Probabilistic Sampling

We begin by reviewing the results with probabilistic sampling. In [1], the authors point out that the formulations (1) or (2) do not always recover an unknown matrix. They illustrate this by taking $X$ as the matrix with a $1$ in the $(1,1)$ position and zeros elsewhere. In this case, unless $(1,1)\in\Omega$, the solution to both (1) and (2) is the zero matrix, which does not equal $X$. The difficulty in this case is that the matrix has high “coherence,” as defined next.

Definition 1**.**

Suppose $X\in{\mathbb{R}}^{n_{r}\times n_{c}}$ has rank $r$ and the reduced singular value decomposition $X=U\Gamma V^{\top}$, where $U\in{\mathbb{R}}^{n_{r}\times r}$, $V\in{\mathbb{R}}^{n_{c}\times r}$, and $\Gamma\in{\mathbb{R}}^{r\times r}$ is the diagonal matrix of the nonzero singular values of $X$. Let $P_{U}=UU^{\top}\in{\mathbb{R}}^{n_{r}\times n_{r}}$ denote the orthogonal projection of ${\mathbb{R}}^{n_{r}}$ onto $U{\mathbb{R}}^{n_{r}}$. Finally, let ${\bf e}_{i}\in{\mathbb{R}}^{n_{r}}$ denote the $i$-th canonical basis vector. Then we define

[TABLE]

where $u^{i}$ is the $i$-th row of $U$. The quantity $\mu_{0}(V)$ is defined analogously, and

[TABLE]

Next, define

[TABLE]

It is shown in [1] that $1\leq\mu_{0}(U)\leq\frac{n_{r}}{r}$. The upper bound is achieved if any canonical basis vector is a column of $U$. (This is what happens with the matrix with all but one element equalling zero.) The lower bound is achieved if every element of $U$ has the same magnitude of $1/\sqrt{n}$, that is, a submatrix of a Walsh-Hadamard matrix.

If one were to sample $m$ out of the $n_{r}n_{c}$ elements of the unknown matrix $X$ without replacement, then one is guaranteed that exactly $m$ distinct elements of $X$ are measured. This is the approach adopted in [1]. However, the disadvantage is that the locations of the $m$ samples are not independent, because once the first element has been selected, there are only $n_{r}n_{c}-1$ choices for the second sample, and so on. Thus sampling without replacement requires quite advanced probabilistic analysis.

Theorem 1**.**

(See [1, Theorem 1.1].) Suppose there is a known constant $\mu$ such that $\mu\geq\max\{\mu_{0}(X),\mu_{1}(X)\}$. Draw

[TABLE]

samples from $[n_{r}]\times[n_{c}]$ without replacement, with a uniform distribution. Then with probability atleast $1-\zeta$ where

[TABLE]

the recovered matrix $\hat{X}$ using (2) is the unique solution. Here $C,c$ are some universal constants that depend on $\mu$, and $n=\max(n_{r},n_{c})$.

An alternative is to sample the elements of $X$ with replacement. In this case the locations of the $m$ samples are indeed independent. However, the price to be paid is that, with some small probability, there would be duplicate samples, so that after $m$ random draws, the number of elements of $X$ that are measured could be smaller than $m$. This is the approach adopted in [15]. On balance, the approach of sampling with replacement is easier to analyze.

Theorem 2**.**

(See [15, Theorem 2].) Assume without loss of generality that $n_{r}\leq n_{c}$. Choose some constant $\beta>1$, and draw

[TABLE]

samples from $[n_{r}]\times[n_{c}]$ with replacement. Define $\hat{X}$ as in (2). Then, with probability at least equal to $1-\zeta$ where

[TABLE]

the true matrix $X$ is the unique solution to the optimization problem, so that $\hat{X}=X$.

A recent result published in [14] gives an improved sufficient condition that involves only the coherence parameter $\mu_{0}(X)$, but not $\mu_{1}(X)$, as in Theorems 2. This approach reduces the number of measurements $m$ by a factor of $r$, but adds a factor of $\log n$. Therefore, whenever $r>\log n$, the approach of [14] is an improvement.

Theorem 3**.**

(See [14, Theorem 1]) Draw

[TABLE]

samples from $[n_{r}]\times[n_{c}]$ uniformly at random, with replacement. Then with probability at least $1-\zeta$ where

[TABLE]

the recovered matrix $\hat{X}$ using (2) is the unique solution. Here $n=\max(n_{r},n_{c})$ and $C,c_{1},c_{2}$ are some constants greater than zero.

Theorems 1-2 gives conditions of exact recovery using the probabilistic method when there are no measurement errors involved. For the case of unstructured noisy measurements (that is, stable matrix completion), the following result is relevant.

Theorem 4**.**

(See [13, Theorem 7].) Draw samples uniformly from $[n_{r}]\times[n_{c}]$ without replacement. Then with probability atleast $1-\zeta$ where

[TABLE]

the recovered matrix $\hat{X}$ using (3) satisfies

[TABLE]

Here $c,C$ are constants dependent on $\mu$.

2.2 Robust Matrix Completion

In this subsection we present a very brief review of robust matrix completion. The interested reader is directed to the references of the papers discussed here for more information.

In [22], the authors analyze the case of noise-free measurements. They begin by observing that $X$ has rank $r$, then the columns of $X$ span an $r$-dimensional subspace, call it ${\mathbb{S}}$. Each measurement leads to some constraints on ${\mathbb{S}}$. By analyzing the structure of the measurements, it may be possible to determine the subspace ${\mathbb{S}}$ uniquely. The following result gives the flavor of the results in [22].222For the convenience of the reader, we use the notation in that paper as opposed to the current notation.

Theorem 5**.**

(See [22, Theorem 3]. Suppose $X\in{\mathbb{R}}^{d\times N}$ and has rank $r$ or less. Suppose $r\leq d/r$, and that each column of $X$ is observed in at least $l$ rows, distributed uniformly and independently across columns, where

[TABLE]

Then with probability at least $1-\epsilon$, the matrix $X$ can be exactly completed.

The “completion algorithm” is not nuclear norm minimization as in earlier papers. Rather, one has to solve a set of polynomial equations; see [22, Section IV-B]. Thus, while the sample complexity of the number of measurements grows slowly, the computational complexity of implementing the recovery algorithm is higher than with nuclear norm minimization.

Note that there is no measurement noise in the formulation of [22]. Since the results in [22] depend on determining which $r$-dimensional subspaces are compatible with the observations, in principle it is possible for the recovery algorithm to be robust against a limited number of erroneous measurements. This is the problem of robust matix completion. One of the recent contributions on this topic is [20], which also contains a wealth of references. The key results are [20, Theorem 4] and [20, Theorem 5]. We do not state those theorems here because of the need to introduce a great deal of notation and background material; instead we refer the reader to the original source.

Another direction of research is tensor completion instead of matrix completion. At a very basic level, one can think of a tensor as a real number indexed by more than two integer indices. However, tensor analysis is far more intricate compared to matrix analysis, in terms of canonical forms, representation, rank factorization etc. The notion of “rank” extends to tensors. Therefore it is reasonable to pose the tensor completion problem as that of reconstructing a tensor from measuring some of its components. A recent paper [18] builds on the general approach proposed in [22] by examining which set of tensors is compatible with a particular set of measurements, and then enlarging the set of measurements until only the true but unknown tensor is compatible with the measurements.

We conclude this brief review by mentioning [19], which introduces the notion of reconstructing a single element of the unknown matrix, and then building on that. One noteworthy feature of this paper is the explicit recognition of the sample set $\Omega$ as the edge set of a bipartite graph. However, unlike with Ramanujan graphs, the approach does not take into account the spectral properties of this bipartite graph.

2.3 Alternatives to Nuclear Norm Minimization

While nuclear norm minimization is the most popular approach to matrix completion, there are other approaches, of which just a few are reviewed here.

In [23], the matrix completion problem is solved via an algorithm called “OptSpace,” which incorporates three steps: Trimming, projecting, and cleaning. The algorithm can be viewed as optimization on the Grassmanian manifold of subspaces. In [24], the OptSpace algorithm is extended to encompass measurement noise.

One approach, inspired by [25], is to enforce the constraint that ${\rm{rank}}(X)\leq r$ by factoring $X$ as $GH^{\top}$, where $G\in{\mathbb{R}}^{n_{r}\times r},H\in{\mathbb{R}}^{n_{c}\times r}$. Let $S=E_{\Omega}\circ X$ denote the measured matrix as above. Then (1) can be reformulated as

[TABLE]

where $\|\cdot\|_{F}$ denotes the Frobenius norm of a matrix, that is, the $\ell_{2}$-norm of its vector singular values, and $\lambda$ is an adjustable weight. See [10, Eq. (10)] for more explanation. The potential difficulty with (16) is that it is no longer a convex optimization problem, and thus could have spurious local minima. Over the years, several papers, including the recent paper [26], show that there are no spurious local minima. Moreover, this statement is true for more general problems than (16), when the quadratic cost function is replaced by more general cost functions.

In [27], the authors first convert the constrained optimization problem (2) to the regularized problem

[TABLE]

where, as before $S=E_{\Omega}\circ X$ is the measured matrix. Then they replace the term $\|Z-S\|_{F}^{2}$ by what they call a correntropy term, as follows:

[TABLE]

where $\sigma$ is an adjustable parameter.

In cases where there is more structure to the unknown matrix $X$, other approaches are possible. In [28], the authors study the problem of robust compressed sensing, and formulate it as a structured matrix completion problem. The problem is then solved via a new algorithm called Enhanced Matrix Completion. In [29], the objective is to estimate an unknown covariance matrix using rank one measurements. Thus $X$ is assumed to be symmetric and positive definite, which is then probed via measurements of the form

[TABLE]

where $a_{l}$ is the probing vector, $\eta_{l}$ is measurement noise, and $m$ is the number of measurements. Note that for an arbitrary $X\in{\mathbb{R}}^{n_{r}\times n_{c}}$, $i\in[n_{r}]$ and $j\in[n_{c}]$, we can write

[TABLE]

where ${\bf e}$ denotes an elementary unit vector. Thus (17) can be viewed as probing $X$ by a more general rank one matrix. Earlier, in [30], it is proposed to use rank one measurements of the form

[TABLE]

where $a_{l},b_{l}$ are random Gaussian vectors, and $X$ is a low-rank matrix. Again, (17) can be viewed as a specialization of the general rank one probe to the case of symmetric (and positive semidefinite) matrices. Note that, in the numerical examples in [28], the vectors $a_{l}$ are random Gaussian or Bernoulli vectors. Finally, in [31], the unknown matrix $X$ is either a single-channel or a multi-channel Hankel matrix.

2.4 Basic Concepts from Graph Theory

The results presented here make use of the concept of Ramanujan graphs. Hence we introduce this concept using a bare minimum of graph theory. Further details about Ramanujan graphs can be found in [32, 33].

Suppose a graph has $n$ vertices, so that its adjacency matrix $A$ belongs to $\{0,1\}^{n\times n}$. Such a graph is said to be $d$-regular if every vertex has degree $d$. Clearly, this is equivalent to saying that every row of $A$ contains precisely $d$ ones (and also every column, because $A$ is symmetric). It is easy to show that $\lambda_{1}=d$ is the largest eigenvalue of $A$. The graph is said to be a Ramanujan graph if the second largest eigenvalue $\lambda_{2}$ satisfies the bound

[TABLE]

The significance of the bound in (18) arises from the so-called Alon-Boppana bound [34, 35], which states the following: If $d$ is kept fixed and $n$ is increased, then the right side of (18) is the best possible bound for $|\lambda_{2}|$.

Suppose $B\in\{0,1\}^{n_{r}\times n_{c}}$. Then $B$ can be interpreted as the biadjacency matrix of a bipartite graph with $n_{r}$ vertices on one side and $n_{c}$ vertices on the other. If $n_{r}=n_{c}$, then the bipartite graph is said to be balanced, and is said to be unbalanced if $n_{r}\neq n_{c}$. The prevailing convention is to refer to the side with the larger ($n_{c}$) vertices as the “left” side and the other as the “right” side. A bipartite graph is said to be left-regular with degree $d_{c}$ if every left vertex has degree $d_{c}$, and right-regular with degree $d_{r}$ if every right vertex has degree $d_{r}$. It is said to be $(d_{r},d_{c})$-biregular if it is both left- and right-regular with row-degree $d_{r}$ and column-degree $d_{c}$. Obviously, in this case we must have that $n_{r}d_{r}=n_{c}d_{c}$. It is convenient to say that a matrix $B\in\{0,1\}^{n_{r}\times n_{c}}$ is “$(d_{r},d_{c})$-biregular” to mean that the associated bipartite graph is $(d_{r},d_{c})$-biregular.

If $B$ is the biadjacency matrix of a bipartite graph with $n_{r}\leq n_{c}$, then the full adjacency matrix of the graph with $n_{r}+n_{c}$ vertices looks like

[TABLE]

It is easy to verify that the eigenvalues of $A$ are $\pm\sigma_{1},\ldots,\pm\sigma_{n_{r}}$ together with an appropriate number of zeros, where $\sigma_{1},\ldots,\sigma_{n_{r}}$ are the singular values of $B$ (some of which could be zero). Moreover, it is easy to show that $\sigma_{1}=\sqrt{d_{r}d_{c}}$. Now, The bipartite graph corresponding to $B$ is defined to be an asymmetric Ramanujan graph if

[TABLE]

where $\sigma_{2}$ represents second largest singular value of the matrix $B$. As with the bound in (18), it can be shown [36] that the right side of (19) is the best possible bound for $\sigma_{2}$. Throughout this paper we will represent $\sigma_{1}$ as the largest singular value and $\sigma_{2}$ as the second largest singular value of the measurement matrix $E_{\Omega}$.

There are relatively few explicit constructions of Ramanujan graphs. The present authors have given a new construction in [37]. Note that, according to [38], Ramanujan graphs of all degrees $d$ and all vertex set sizes $n$ exist, though as yet there are no efficient algorithms for finding them. See [39] for some preliminary results in this direction.

2.5 A Claimed Previous Result

In this section we present a claimed result from [17] on matrix completion using a Ramanujan graph to generate the sampling set. To facilitate the statement of these results, we reproduce two standard coherence assumptions on the unknown matrix $X=U\Gamma V^{\top}$. As is standard, we use $A^{i},A_{j},A_{ij}$ to denote the $i$-th row, the $j$-th column, and the $(i,j)$-th element of a matrix $A$. We also use $\|A\|_{S}$ to denote the spectral or operator norm of the matrix $A$, that is, the largest singular value of $A$.

1. (A1).

There is a known upper bounds $\mu_{0}(X)$ on $\mu_{0}(U)$ and $\mu_{0}(V)$ respectively. Hereafter simply write $\mu_{0}$ for $\mu_{0}(X)$. 2. (A2).

There is a constant $\theta$ such that

[TABLE]

[TABLE]

where $U^{k\top}$ is shorthand for $(U^{k})^{\top}$, $V^{k\top}$ is shorthand for $(V^{k})^{\top}$ and $d_{r},d_{c}$ are the degrees of the Ramanujan bigraph $\Omega$. If Ramanujan graph is a balanced graph, then $n_{r}=n_{c}=n$ and $d_{r}=d_{c}=d$.

The following result is claimed in [17].

Theorem 6**.**

(See [17, Theorem 4.2].) Suppose Assumptions (A1) and (A2) hold. Choose $E_{\Omega}$ to be the adjacency matrix of $d$ regular graph such that $\sigma_{2}(E_{\Omega})\leq C\sqrt{d}$, and $\theta<1/6$. Define $\hat{X}$ as in (2). With these assumptions, if

[TABLE]

Then the true matrix $X$ is the unique solution to the optimization problem (2). In particular, if the graph is a Ramanujan graph, then $C=2$, so that (22) becomes

[TABLE]

However, there is one step in the proffered proof of the above theorem that does not appear to be justified. More details are given in the Appendix.

3 New Results

3.1 Theorem Statement

In this section we state, without proof, the main result of the paper, and discuss its implications. The proof of the theorem makes use of some preliminary results, which are stated and proved in Section 4. The proof of the main theorem is given in Section 5.

To avoid repeating the same text over and over, we introduce the following standing notation: Suppose $X\in{\mathbb{R}}^{n_{r}\times n_{c}}$ is a matrix of rank $r$ or less, and satisfies the incoherence assumptions $A1$ and $A2$ with constants $\mu_{0}$ and $\theta$.333Note that, unlike [1, 15], we do not require the constant $\mu_{1}$. Suppose $E_{\Omega}\in\{0,1\}^{n_{r}\times n_{c}}$ a biadjacency matrix of a $(d_{r},d_{c})$ biregular graph $\Omega$, and let $\sigma_{1},\sigma_{2}$ denote the first and second largest singular value of the matrix $E_{\Omega}$. Next, define

[TABLE]

and note that $\alpha$ is the fraction of elements of $X$ that are being sampled. Finally, define

[TABLE]

and observe that $\phi$ depends on two distinct pieces of information: The ratio $\sigma_{2}/\sigma_{1}$ which depends on the sampling matrix $E_{\Omega}$, and the quantity $\mu_{0}$ which depends on the unknown matrix $X$.

Theorem 7**.**

With the standing notation, suppose without loss of generality that $n_{r}\leq n_{c}$. Suppose that $\theta+\phi<1$, and that

[TABLE]

Define constants $c,\gamma$ as

[TABLE]

[TABLE]

and note that $c>0$ as a consequence of (26). Then every solution $\hat{X}$ of (3) satisfies, for every $\delta>0$, the bound

[TABLE]

Corollary 1**.**

If the hypotheses of Theorem 7 hold, then $\hat{X}=$ is the unique solution of (2).

Proof.

Substitute $\epsilon=0$ in (29). ∎

Now we compare Theorem 7 with the prior literature discussed in Section 2.

Compared to Theorems 1, 2 and 3, the conclusions of Theorem 7 always hold, and not just with high probability. 2. 2.

Unlike Theorems 1 and 2 (but like Theorem 3), our result makes use of only the coherence constant $\mu_{0}$, but not $\mu_{1}$. 3. 3.

So far as the authors are aware, the only other result that applies to the case of non-sparse noisy measurements is Theorem 4. That is also a probabilistic result, while Theorem 7 is deterministic.

3.2 Sample Complexity Analysis

Now we attempt to compare the number of measurements required by our approach with the number required by earlier theorems in Section 2. A direct comparison is difficult due to the following factors. Theorems 1 and 3 are stated in terms of universal constants that are not explicitly computable. Therefore these two theorems can be used only to bound the rate of growth of the number of measurements $m$. In Theorem 2, the bound for $m$ is quite explicit. However, it involves both coherence constants $\mu_{0}$ and $\mu_{1}$, while our bound (as also those in Theorems 3 and 6) do not involve the coherence parameter $\theta_{1}$.

Note that, in Theorem 7, $\alpha$ is the fraction of elements of $X$ that are measured, so that the total number of measurements is $\alpha n_{r}n_{c}$. Clearly the smaller the value of $\alpha$ given by (26), the fewer the number of measurements. Therefore, for a like to like comparison, we compare (26) with that in (23), even though the proof of Theorem 6 contains a gap, as shown in the Appendix.

To facilitate the comparison, let us restrict to square matrices, so that $d_{r}=d_{c}=d$, and $n_{r}=n_{c}=n$. Choose the coherence parameter $\mu_{0}$ as $1.5$, which is only slightly larger than the theoretical minimum value of $1$. It is always the case that $\sigma_{1}=d$. If the graph is a Ramanujan graph, it can be assumed that $\sigma_{2}=2\sqrt{d}$ after neglecting the $-1$ term. Therefore the ratio $\sigma_{2}/\sigma_{1}$ equals $2/\sqrt{d}$. Finally let $r=2$, which is very small (but comparable to the range of values for which Theorems 1 and 2 apply). With these numbers, we have that $\mu_{0}r=3$. Hence (23) gives

[TABLE]

Therefore, by applying Theorem 6, one would have to measure $1,296$ elements of $X$ in each row and each column. Clearly the degree $d$ cannot exceed the number of vertices $n$. Therefore (leaving aside the validity of (23)), Theorem 6 does not apply unless $n\geq 1,296$.

Now let us apply Theorem 7. By definition

[TABLE]

Let us choose $\theta=0.1$. If we set $d=800$ which is lower than the number from Theorem 6, then the lower bound in (26) becomes

[TABLE]

However, if we decrease $d$ to $500$, then we get

[TABLE]

In either case $\alpha<1$. Therefore Theorem 7 is applicable with fewer measurements per row and column than the claimed Theorem 6.

Note that the size of the matrix $n$ does not enter the bound (26). It is easy to see that the total number of measurements is

[TABLE]

or exactly $\max\{d,\alpha n\}$ measurements per row. So if $n=2,000$, both numbers of roughly equal. If the matrix size becomes larger, then $\alpha$ is the decisive factor and not $d$.

4 Preliminary Results

In this section we state and prove various preliminary results that are used in the proof of Theorem 7, which is given Section 5.

The result below is used repeatedly to analyze triple products.

Lemma 1**.**

Suppose $M\in{\mathbb{R}}^{n_{r}\times n_{c}}$, $A\in{\mathbb{R}}^{n_{r}\times r}$, and $B\in{\mathbb{R}}^{n_{c}\times r}$. Suppose further that $x\in{\mathbb{R}}^{n_{r}},y\in{\mathbb{R}}^{n_{c}}$. Then

[TABLE]

Proof.

The proof follows readily by expanding the triple product. Note that

[TABLE]

Therefore

[TABLE]

as desired. ∎

Lemma 2**.**

Suppose $A\in{\mathbb{R}}^{n_{r}\times r},z\in{\mathbb{R}}^{n_{r}}$, and suppose further that

[TABLE]

Then

[TABLE]

Proof.

By definition

[TABLE]

as desired. ∎

Lemma 3**.**

Suppose $M,A,B$ are as in Lemma 1. Suppose further that

[TABLE]

Then

[TABLE]

Proof.

Recall that, for any matrix $X$, we have that

[TABLE]

In particular

[TABLE]

where the last step follows from Lemma 1. Now fix $x,y$ such that $\|x\|_{2}=1,\|y\|_{2}=1$. Then

[TABLE]

Next, apply Schwarz’s inequality to deduce that

[TABLE]

where the last step follows from applying Lemma 2 twice, and observing that $\|x\|_{2}=1,\|y\|_{2}=1$. Substituting this bound in (35) shows that

[TABLE]

whenever $\|x\|_{2}=1,\|y\|_{2}=1$, which is the desired result. ∎

Now we introduce a special matrix $M$, which plays a central role in solving the matrix completion problem.

Lemma 4**.**

Define

[TABLE]

Then $\|M\|_{S}=\sigma_{2}/\alpha$.

Proof.

Because $E_{\Omega}$ is biregular, it follows that $\sigma_{1}=\sqrt{d_{r}d_{c}}$ is the largest singular value of $E_{\Omega}$. Moreover, $(1/\sqrt{n_{r}}){\bf 1}_{n_{r}}^{\top}$ is a row singular vector of $E_{\Omega}$, and $(1/\sqrt{n_{c}}){\bf 1}_{n_{c}}$ is a column singular vector. Therefore a SVD of $E_{\Omega}$ looks like

[TABLE]

where the largest singular value of $B$ is $\sigma_{2}$. The desired bound follows by observing that $M=(1/\alpha)B$. ∎

Theorem 8**.**

Subject to the standing notation, we have that

[TABLE]

where $\phi$ is defined in (25), and $\|\cdot\|_{S}$ denotes the spectral norm (largest singular value) of a matrix.

Remark:

• •

Note that Lemma 1 is essentially the same as [17, Theorem 4.1]. However, our proof is much simpler.

• •

Observe that ${\bf 1}_{n_{r}\times n_{c}}\circ X=X$ for all $X$. Therefore

[TABLE]

Therefore Theorem 8 states that the sampled matrix $E_{\Omega}\circ X$, normalized by the factor $1/\alpha$ for the fraction of elements sampled, can provide an approximation to $X$. The smaller the constant $\phi$, the better the approximation.

Proof.

Now suppose $X=U\Gamma V^{\top}$ is a singular value decomposition of $X$, where $\Gamma={\rm Diag}(\gamma_{1},\ldots,\gamma_{r})$. Define $A=U\Gamma,B=V$. Then $X=AB^{\top}$. Moreover

[TABLE]

because $\|X\|_{S}=\gamma_{1}$, and the definition of the coherence $\mu_{0}$. Similarly

[TABLE]

Now apply Lemma 3 with

[TABLE]

and note that $\alpha\sqrt{n_{r}n_{c}}=\sqrt{d_{r}d_{c}}=\sigma_{1}$. Then (34) becomes

[TABLE]

as desired. ∎

Lemma 5**.**

Suppose $E_{\Omega}\in\{0,1\}^{n_{r}\times n_{c}}$ is a $(d_{r},d_{c})$-biregular sampling matrix, and let $U,V,\theta$ be as before.

For arbitrary $B\in{\mathbb{R}}^{n_{c}\times r}$, define

[TABLE]

Then

[TABLE]

[TABLE] 2. 2.

For arbitrary $C\in{\mathbb{R}}^{n_{r}\times r}$, define

[TABLE]

Then

[TABLE]

[TABLE]

Proof.

We begin by proving (40). Equation (41) is a direct consequence of (40)

In view of the definition of the matrix $M$, (39) can be rewritten as

[TABLE]

Fix $i\in[r],j\in[n_{c}]$. Then

[TABLE]

Let us focus on the first term after ignoring the factor of $1/\alpha$. From Lemma 1, specifically (30), we get

[TABLE]

Now observe that $B_{k}\circ{\bf e}_{j}=B_{jk}{\bf e}_{j}$, so that $E_{\Omega}(B_{k}\circ{\bf e}_{j})=(E_{\Omega})_{j}B_{jk}{\bf e}_{j}$. Therefore

[TABLE]

For this fixed $j$, define

[TABLE]

and note that $|{\cal N}(j)|=d_{c}$ due to regularity. Then, for fixed $k\in[r]$, we have

[TABLE]

Therefore

[TABLE]

By (20), the matrix inside the square brackets has spectral norm $\leq\theta$. Therefore

[TABLE]

which is (40). Taking the norm squared and summing over all $j$ proves (41), after noting that a matrix and its transpose have the same Frobenius norm. This establishes Item (1).

To prove Item (2), we use Item (1). Note that $(X\circ Y)^{\top}=X^{\top}\circ Y^{\top}$. So (42) is equivalent to

[TABLE]

Now every column of $E_{\Omega}^{\top}$ (or every row of $E_{\Omega}$) contains $d_{r}$ ones. Therefore (43) follows from (40), and (44) follows from (41). ∎

Next, define ${\cal T}\subseteq{\mathbb{R}}^{n_{r}\times n_{c}}$ to be the subspace spanned by all matrices of the form $UB^{\top}$ and $CV^{\top}$. It is easy to show that the projection operator ${\cal P}_{\cal T}$ equals

[TABLE]

where $U_{\perp},V_{\perp}$ are chosen such that such that $[U\;\;U_{\perp}]$ and $[V\;\;V_{\perp}]$ are square and orthogonal. This ensures that $U_{\perp}U_{\perp}^{\top}=I_{n_{r}}-UU^{\top}$ and $V_{\perp}V_{\perp}^{\top}=I_{n_{c}}-VV^{\top}$.

The next two lemmas characterize the map ${\cal T}\rightarrow{\cal T}$ defined by $Z\mapsto(1/\alpha)P_{\cal T}E_{\Omega}\circ Z-Z$. Note that this is not quite the map $M\circ Z$, because of the presence of the term ${\cal P}_{\cal T}$.

Lemma 6**.**

Suppose $E_{\Omega}\in\{0,1\}^{n_{r}\times n_{c}}$ is a $(d_{r},d_{c})$-biregular sampling matrix, and that $Z\in{\cal T}$. Define

[TABLE]

so that $Z=UB^{\top}+CV^{\top}$. Next, define

[TABLE]

[TABLE]

Then

[TABLE]

[TABLE]

Remark: The above two relations can be expressed compactly as

[TABLE]

Proof.

We establish (48), and the proof of (49) is entirely similar.

The definition of ${\cal P}_{\cal T}$ makes it clear that

[TABLE]

Therefore

[TABLE]

Define $\bar{B}^{\top}=\bar{B}^{\top}_{1}+\bar{B}^{\top}_{2}$, where

[TABLE]

Then it follows from Lemma 5 that

[TABLE]

To estimate $\|\bar{B}_{2}\|_{F}=\|\bar{B}_{2}^{\top}\|_{F}$, we proceed as follows:

[TABLE]

[TABLE]

Fix a $y\in{\mathbb{R}}^{n_{c}}$ such that $\|y\|_{2}=1$ but otherwise arbitrary, and define

[TABLE]

Then it follows by Lemma 1 that

[TABLE]

where we use Schwarz’s inequality in the last step. Now observe that, by the definition of the coherence $\mu_{0}$, we have

[TABLE]

Therefore it follows from Lemma 2 that

[TABLE]

because $\|y\|_{2}^{2}=1$. Next, from Lemma 2 and

[TABLE]

it follows that

[TABLE]

Combining both bounds gives

[TABLE]

after noting that $\alpha\sqrt{n_{r}n_{c}}=\sigma_{1}$. Taking square roots of both sides gives

[TABLE]

Finally

[TABLE]

which is (48). The proof of (49) is entirely similar. ∎

Lemma 7**.**

Suppose $E_{\Omega}\in\{0,1\}^{n_{r}\times n_{c}}$ is a $(d_{r},d_{c})$-biregular sampling matrix, that $Z\in{\cal T}$, and as in (46), define

[TABLE]

Then

[TABLE]

where $\theta,\phi$ are defined in (48) and (49) respectively.

Remark: The above lemma can be stated as follows: The map $Z\mapsto(1/\alpha)P_{\cal T}E_{\Omega}\circ Z-Z$, when restricted to ${\cal T}$, has an operator norm $\leq\theta+\phi$.

Proof.

Define, as before,

[TABLE]

[TABLE]

so that $Z=UB^{\top}+CV^{\top}$, $\bar{Z}=U\bar{B}^{\top}+\bar{C}V^{\top}$. Note that

[TABLE]

because $U^{\top}C=0$. Therefore

[TABLE]

because left multiplication by $U$ and right multiplication by $V^{\top}$ preserve the Frobenius norm. Similarly

[TABLE]

Now it is easy to verify that the spectral norm of the matrix in (50) is $\theta+\phi$. Therefore

[TABLE]

This is the desired conclusion. ∎

5 Proof of Theorem 7

The proof of Theorem 7 depends on the following auxiliary lemma, which is reminiscent of the “dual certificate” condition that is widely used in solving the matrix completion problem; see for example [15, Theorem 2]. It should be noted that Lemma 8 provides a framework for a solution, which we then specialize, somewhat inefficiently, in Theorem 7. Finding better ways to apply Lemma 8 is a problem for future research.

Let $X=U\Gamma V^{\top}$ be a reduced SVD of $X$. Throughout, we use the symbols ${\cal T},{\cal P}_{\cal T},{\cal P}_{{\cal T}^{\perp}},U_{\perp},V_{\perp}$ introduced in the previous section.

Lemma 8**.**

Suppose there exists a matrix $Y\in{\mathbb{R}}^{n_{r}\times n_{c}}$ such that $E_{\Omega}\circ Y=Y$, and constants $k_{1},k_{2}\in(0,1),k_{3}>0$ such that

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

Define the constant

[TABLE]

Then every solution $\hat{X}$ of (3) satisfies the bound

[TABLE]

Proof.

(Of Lemma 8.) Observe that

[TABLE]

Define $H=\hat{X}-X$, so that $\hat{X}=X+H$. Also, it can be assumed without loss of generality that ${\cal W}$ is supported on $\Omega$, so that $E_{\Omega}\circ{\cal W}={\cal W}$ and $E_{\Omega^{c}}\circ{\cal W}=0$. Now we can write

[TABLE]

because (i) $\hat{X}$ is feasible for (3), and thus $\|E_{\Omega}\circ(\hat{X}-X-{\cal W})\|_{F}\leq\epsilon$ (see (3)), and $\|W\|_{F}\leq\epsilon$. Next,

[TABLE]

Therefore, once we are able to find an upper bound for $\|E_{\Omega^{c}}\circ H\|_{F}$, the above relationship leads to an upper bound for $\|H\|_{F}=\|\hat{X}-X\|_{F}$.

Define $H_{\Omega}=E_{\Omega}\circ H$ and $H_{\Omega^{c}}=E_{\Omega^{c}}\circ H$. Then

[TABLE]

Next, write $H_{\Omega^{c}}={\cal P}_{\cal T}(H_{\Omega^{c}})+{\cal P}_{{\cal T}^{\perp}}(H_{\Omega^{c}})$, and note that $\langle Y,H_{\Omega^{c}}\rangle_{F}=0$ because $Y$ is supported on $\Omega$ and $H_{\Omega^{c}}$ is supported on $\Omega^{c}$. Now observe that, for any matrix $B\in{\mathbb{R}}^{n_{r}\times n_{c}}$, we have that

[TABLE]

Therefore

[TABLE]

where $(a)$ follows from $\langle Y,H_{\Omega^{c}}\rangle_{F}=0$, and $(b)$ follows from (64). Using (63) and (5) together we get

[TABLE]

On the other hand, $\|\hat{X}\|_{N}\leq\|X\|_{N}$ because $X$ is feasible for the problem in (3), and $\hat{X}$ is a solution of (3). Substituting this into (66), cancelling $\|X\|_{N}$ from both sides, and rearranging terms, gives

[TABLE]

Now,

[TABLE]

where $(a)$ follows from ${\cal P}_{\cal T}^{2}={\cal P}_{\cal T}$, and $(b)$ follows from assumption (56). Next, note that $E_{\Omega}\circ H_{\Omega^{c}}=0$, which in turn implies that

[TABLE]

so that

[TABLE]

Therefore

[TABLE]

where the last step follows from (5). Now using (5) in (67) gives us

[TABLE]

where we use the assumptions (55) and (58), together with (5). Next

[TABLE]

where $(a)$ follows from (5), $(b)$ follows from the fact that $H_{\Omega}\in{\mathbb{R}}^{n_{r}\times n_{c}}$ and thus has rank no more than $\min\{n_{r},n_{c}\}=n_{r}$, and $(c)$ follows from (61). Using (5) we get

[TABLE]

where, as above, $(a)$ follows from the fact $E_{\Omega}\circ H_{\Omega^{c}}=0$. Now (62) can be written as

[TABLE]

where $(a)$ follows from (5) and $(b)$ follows from (5). It now follows that

[TABLE]

which is (60). ∎

Proof.

(Of Theorem 7) At last we come to the proof of the main theorem. Suppose $\theta+\phi<1$ (which automatically implies that $\phi<1$), and define

[TABLE]

The proof consists of showing that, under the stated hypotheses, there exists a matrix $Y\in{\mathbb{R}}^{n_{r}\times n_{c}}$ that satisfies the hypotheses of Lemma 8.

We start with (56). Lemma 7 states the following: If $Z\in{\cal T}$ and $\bar{Z}:=(1/\alpha){\cal P}_{T}E_{\Omega}\circ Z-Z$, then

[TABLE]

where $\theta$ is defined in (20) and $\phi$ is defined in (25). Therefore (56) is satisfied with $k_{2}=\theta+\phi$. Next, let $W_{0}:=UV^{\top}\in{\cal T}$, and define

[TABLE]

Then

[TABLE]

Now we can write $W_{0}=UB^{\top}$ where $B=V$, and apply Lemma 6. This gives

[TABLE]

Finally, observe that, because $W_{0}\in{\cal T}$, we can reason as follows:

[TABLE]

Now we can apply Lemma 3, with

[TABLE]

This gives

[TABLE]

Hence we can choose $k_{1}$. The theorem now follows from applying Lemma 8 ∎

There is considerable scope for improving Theorem 7. As shown earlier, we can take $k_{2}=\theta+\phi$. Define $W_{0}=UV^{T}$ as before, and define $W_{i}$ as

[TABLE]

[TABLE]

Then $E_{\Omega}\circ Y_{p}=Y_{p}$ for every integer $p$. To establish (57), note that

[TABLE]

So

[TABLE]

Therefore

[TABLE]

If $\theta+\phi<1$, then (57) holds for $p$ sufficiently large, no matter what $k_{3}$ is. The difficulty however is to find a constant $k_{1}$ such that (55) is satisfied. This is an object of ongoing research by the authors.

6 Phase Transition Studies

The bounds in Theorem 7 provide sufficient conditions for matrix completion using nuclear norm minimization, when the sample set is chosen as the edge set of a Ramanujan graph or a Ramanujan bigraph. These results are only sufficient conditions. It is possible to determine how close these sufficient conditions are to being necessary via numerical simulations. That is the objective of the present section. Our simulations show that choosing $d\approx 3r$ seems to suffice to recover randomly generated matrices of rank $r$. Thus $r_{0}\approx d/3$ is the critical value below which the success ratio is 100% for recovering randomly generated matrices of rank $r\leq r_{0}$. Moreover, the simulations show the presence of a “phase transition,” whereby the likelihood of success falls sharply from 100% to 0% as $r$ is increased by just $2$ or $3$ above the critical value $r_{0}$. These observations are consistent with previous numerical studies on phase transition. Our numerical results are presented next, and they are placed in context against earlier results in the subsequent subsection.

6.1 Numerical Experiments

We carried out some numerical experiments on the behavior of nuclear norm minimization for matrix completion, on randomly generated low rank matrices. To construct Ramanujan graphs to choose the sample matrices, we used the so-called LPS construction proposed in [40]. This construction is based on choosing two prime numbers $p,q$ both congruent to $1~{}{\rm mod}~{}4$. The resulting graph has $(q(q^{2}-1))/2$ vertices and degree $p+1$. In the original construction, $p<q$. However, it is possible to choose $p>q$ provided some consistency conditions are satisfied. The authors have written Matlab code that implements the LPS construction; this code is available from the authors upon request.

For illustrative purposes, we chose $q=13$, which leads to graphs with $n=1,092$ vertices. By varying the prime number $p$, we could generate Ramanujan graphs with varying degrees. Every prime number equal to $1~{}{\rm mod}~{}4$ between $5$ and $157$ results in a Ramanujan graph using the LPS construction. However, still larger choices of $p$ are permissible, such as $197,229$ and $293$. However, for some intermediate values of $p$ between $157$ and $197$, the resulting LPS construction would have multiple edges between some pairs of vertices.

For each choice of $p$, (that is, each choice of the degree $d$), we varied the rank $r$ from 1 upwards, and for each $r$ generated 100 random matrices of dimensions $1092\times 1092$ and rank $r$. Then we applied nuclear norm minimization, and examined what fraction of the 100 rank $r$ matrices were correctly recovered. The criterion for correct recovery was that the normalized Frobenius norm $\|\hat{X}-X\|_{F}/\|X\|_{F}$ was less than $10^{-6}$. However, the results are quite insensitive to this number. The objective was to determine how the critical value of the rank $\bar{r}$, defined as the value below which 100% recovery takes place, depends on $d=p+1$. The results are shown in Figure 1. It can be observed from this figure that $\bar{r}\approx d/3=(p+1)/3$ over the entire range of $p$. For comparison purposes, we repeated the study, with the Ramanjuan graph sample set replaced by randomly chosen samples. There is very little difference between the two curves. In a sense this is not surprising, because Ramanujan graphs replicate many of the desirable properties of random graphs, including expansion properties. However, the Ramanujan construction provides a systematic method for selecting the sample set.

More interesting was the fact that, if $r$ was increased by just one two above the critical value $\bar{r}$, the percentage of accurately recovered matrices fell sharply from 100% to 0%. This is illustrated in Figure 2 for $p=197$ or $d=198$, in Figure 3 for $p=229$ of $d=230$, and in Figure 4 for $p=293$ or $d=294$. This phenomenon is known as “phase transition,” and is well-known for vector recovery using $\ell_{1}$-norm minimization. However, so far as we are aware, a similar phenomenon has not been reported for matrix completion using nuclear norm minimization.

6.2 Comparison with Previous Studies

The phenomenon of phase transition is well-known in the case of vector recovery using $\ell_{1}$-norm minimization. It is also well-understood, and is discussed in several papers by David Donoho, such as [41, 42, 43, 44, 45, 46, 47, 48]. A very general theory of the behavior of convex optimization on randomly generated data is given in [49]. A concept called the “statistical dimension” is introduced, and it is established that convex optimization exhibits “phase transition,” whereby the success rate of the optimization algorithm goes from 100% to 0% very quickly as the input to the algorithm is changed. The width of the transition region is linear in the statistical dimension. In contrast, there are relatively fewer papers that have studied phase transition in matrix completion, and a proper theory is not yet available. In principle, the theory in [49] applies also to matrix completion using nuclear norm minimization. However, it is not easy to work out the appropriate statistical dimension in this case. In [22], the behaviour of the manifold approach to recovering randomly generated matrices is studied. A related paper is [50], which studies matrix recovery, and not matrix completion. In that paper, the measurements consist of taking the Frobenius inner product of the unknown matrix with randomly generated Gaussian matrices.

Now we describe some of the prior work on phase transition in matrix completion via nuclear norm minimization. In all the studies described below, the unknown matrix $X$ is assumed to be square, so we use the neutral symbol $n=n_{c}=n_{c}$ to denote its size. Throughout, $r$ denotes its rank, and $m$ denotes the number of measurements, that is, $|\Omega|$. This is consistent with the notation in the present paper, though the notation in the papers discussed below is in general different. Note that in some papers, $X$ satisfies additional constraints such as symmetry, positive semidefiniteness, Hankel or Toeplitz structure, etc. In [1], which first introduced this approach, there are some examples based on randomly generated low-rank matrices with $n=40$ or $n=50$. In [23], $n=500$, and $r\in\{10,20,40\}$. In [28], $n=11$ or $n=15$. In [29], $n=50$. In [31], $n=300$. Finally, in [27], $n=1,000$, and $r/n\in[0.04,0.4]$. In all cases, it is observed that the critical rank $r_{0}$ at which the recovery ratio drops sharply from 100% to 0% is essentially linear in $m$, the number of measurements. In our setting, since $m=dn$ where $d$ is the degree of the Ramanujan graph, this translates to $r_{0}$ being linear in $d$, as observed in the next subsection. Thus, to summarize, available numerical studies (including ours), indicate the presence of a phase transition, and a linear relationship between the critical rank $r_{0}$ and the number of measurements $m$.

7 Conclusions and Future Work

In this paper we have studied the matrix completion problem with emphasis on choosing the elements to be sampled in a deterministic fashion. We do this by choosing the sample matrix to equal the biadjacency matrix of an asymmetric Ramanujan graph, or Ramanujan bigraph. We have derived a sufficient condition that guarantees exact recovery of the unknown matrix with noise-free measurements using nuclear norm minimization, and stable recovery under non-sparse noisy measurements. We believe that we are presenting the very first correct result on exact recovery using nuclear norm minimization and a deterministic sampling pattern. An earlier paper [17, Theorem 4.2] claims a similar result, but there is one step in the proof that we believe is not justified. This is elaborated in the Appendix.

The sufficient condition given here is very conservative. It requires that the degree of the Ramanujan graph should be $\Omega(r^{3})$ where $r$ is the rank of the matrix to be recovered. Turning this around, our result implies that given a Ramanujan graph of degree $d$, we can guarantee exact recovery only when $r=O(d^{1/3})$. This naturally raises the question as to how close is the sufficient condition derived here to being necessary? Our numerical simulations have shown that $d\approx 3r$ seems to be sufficient to recover randomly generated matrices of rank $r$. More interesting, if $r$ is increased by just two or three above this critical value of $d/3$, the percentage of the randomly generated matrices that are recovered falls from 100% to 0%, a phenomenon known as “phase transition.”

One of the advantages of the random sampling approach is that it is relatively easy to account for “missing” measurements, by ensuring that the missing location is never sampled. In the present approach, this leads to a very interesting problem in graph theory, namely the construction of Ramanujan graphs when there is a “bar” on having edges between specified pairs of vertices. The authors are exploring this question, which would be of considerable interest to graph theorists, quite apart from the matrix completion researchers.

Appendix

In this appendix we point out an error in the proof of [17, Theorem 4.2], which is based on a recursion lemma [17, Lemma 7.3]. However, the proof of the recursion lemma is not given in [17], but is found in the longer version of the paper in [51]. On page 19 of [51], the third displayed (but unnumbered) equation is as follows:444For the convenience of the reader, we use the notation in [51], as opposed to the current notation. Also, while the equation in question is shown on a single line in [51], it is broken into several lines here to fit into the double-column format.

[TABLE]

where $\hat{u}=U_{\perp}U_{\perp}^{i}$.

Let us focus on the claimed equality highlighted by us as $(a)$. This equation is incorrect. If we represent $U_{\perp}U_{\perp}^{\top}$ as $(I_{n_{r}}-UU^{\top})$, then by orthogonality we have

[TABLE]

which in turn implies that

[TABLE]

However, when $i=j$, we have that

[TABLE]

for all $i$. Therefore

[TABLE]

More elaborately

[TABLE]

Therefore, unless $\|U^{i}\|_{2}^{2}=1/2$ or zero, the step highlighted as $(a)$ in (77) is false for that value of $i$. The conclusion is that, in order for (77) to hold, every row of $U$ must either be identically zero, or have Euclidean norm of $1/2$. A row of $U$ being identically zero makes the corresponding row of the unknown matrix $X$ also identically zero. In short, (78) cannot hold except under extremely restrictive and unrealistic conditions. Similar reasoning is used for $V$ which is also not correct. Hence using this Lemma 7.3 in bounding $\|{\cal P}_{{\cal T}^{\perp}}(Y)\|_{S}$ on page 12 of [51] makes the proof of the Theorem 4.2 incorrect. However, it is of course possible that the theorem itself is correct.

Acknowledgement

The authors thank the Associate Editor and three anonymous reviewers for drawing their attention to several relevant references, and for their constructive criticisms that have greatly enhanced the paper. In addition, the authors thank Prof. Alex Lubotzky of Hebrew University, Jerusalem, Israel for useful hints on the construction of Ramanujan graphs of high degree, and Prof. Cristina Ballantine of the College of the Holy Cross for discussions on the construction of Ramanujan bigraphs.