On the Phase Transition of Corrupted Sensing

TL;DR

This paper provides a theoretical explanation for the sharp phase transition observed in corrupted sensing problems, identifying the threshold where convex recovery methods succeed or fail, supported by numerical validation.

Contribution

It establishes the precise threshold for successful recovery in corrupted sensing, linking it to the Gaussian widths of tangent cones, thus explaining the phase transition phenomenon.

Findings

Sharp phase transition occurs around the sum of Gaussian widths squared.

Theoretical thresholds match numerical experiments.

Convex procedures fail or succeed based on this threshold.

Abstract

In \cite{FOY2014}, a sharp phase transition has been numerically observed when a constrained convex procedure is used to solve the corrupted sensing problem. In this paper, we present a theoretical analysis for this phenomenon. Specifically, we establish the threshold below which this convex procedure fails to recover signal and corruption with high probability. Together with the work in \cite{FOY2014}, we prove that a sharp phase transition occurs around the sum of the squares of spherical Gaussian widths of two tangent cones. Numerical experiments are provided to demonstrate the correctness and sharpness of our results.

Equations

$$\mathbf{y}=\mathbf{\Psi }{\mathbf{x}}^{\star }+{\mathbf{v}}^{\star },$$

$$\min f(\mathbf{x}),\text{s.t. }\mathbf{y}=\mathbf{\Psi }\mathbf{x}+\mathbf{v},g(\mathbf{v})\le g({\mathbf{v}}^{\star }),$$

$$\min g(\mathbf{v}),\text{s.t. }\mathbf{y}=\mathbf{\Psi }\mathbf{x}+\mathbf{v},f(\mathbf{x})\le f({\mathbf{x}}^{\star }).$$

$$\omega (T)=\mathbb{E}\underset{\mathbf{t}\in T}{\sup }⟨\mathbf{g},\mathbf{t}⟩,\text{ where }\mathbf{g}\sim N(0,I_n).$$

$$\mathcal{D}_{s}=\big{\{}\bm{a}\in\mathbb{R}^{n}:\exists~{}t>0,f(\bm{x}^{\star}+\bm{a}t)\leq f(\bm{x}^{\star})\big{\}}.$$

$$\mathcal{D}_{c}=\big{\{}\bm{b}\in\mathbb{R}^{m}:\exists~{}t>0,g(\bm{v}^{\star}+\bm{b}t)\leq g(\bm{v}^{\star})\big{\}}.$$

$$\sqrt{m}<\sqrt{\omega^{2}\big{(}\mathcal{D}_{s}\cap S^{n-1}\big{)}+\omega^{2}\big{(}\mathcal{D}_{c}\cap S^{m-1}\big{)}}-t,$$

$$\sqrt{m}\ge \sqrt{{\omega }^2({\mathcal{D}}_s\cap S^{n-1})+{\omega }^2({\mathcal{D}}_c\cap S^{m-1})}+\frac{1}{\sqrt{2}}+\frac{1}{\sqrt{2\pi }}+t,$$

$$\omega^{2}\big{(}\mathcal{D}_{s}\cap S^{n-1}\big{)}+\omega^{2}\big{(}\mathcal{D}_{c}\cap S^{m-1}\big{)},$$

$$C\sqrt{\omega^{2}\big{(}\mathcal{D}_{s}\cap S^{n-1}\big{)}+\omega^{2}\big{(}\mathcal{D}_{c}\cap S^{m-1}\big{)}},$$

$$\omega^{2}\big{(}\mathcal{D}_{s}\cap S^{n-1}\big{)}+\omega^{2}\big{(}\mathcal{D}_{c}\cap S^{m-1}\big{)}\approx\delta(\mathcal{D}_{s})+\delta(\mathcal{D}_{c})=\delta(\mathcal{D}_{s}\times\mathcal{D}_{c}),$$

$$\mathbf{z}=\mathbf{x}+\mathbf{U}\mathbf{y},$$

$$\mathbf{y}={\mathbf{\Psi }}_0{\mathbf{x}}_0+{\mathbf{\Psi }}_1{\mathbf{x}}_1,$$

$$\min_{(\bm{a},\bm{b})\in(\mathcal{D}_{s}\times\mathcal{D}_{c})\cap S^{n+m-1}}\big{\|}\bm{\Psi}\bm{a}+\bm{b}\big{\|}=0.$$

$$\min_{\|\bm{r}\|=1}\min_{\bm{s}\in(\mathcal{D}_{s}\times\mathcal{D}_{c})^{\circ}}\big{\|}\bm{s}-\bm{A}^{*}\bm{r}\big{\|}>0,$$

$$({\mathcal{D}}_s\times {\mathcal{D}}_c{)}^{\circ }={\mathcal{D}}_s^{\circ }\times {\mathcal{D}}_c^{\circ }.$$

$$\min_{\|\bm{r}\|=1}\min_{\bm{s}\in\mathcal{D}_{s}^{\circ}\times\mathcal{D}_{c}^{\circ}}\big{\|}\bm{s}-\bm{A}^{*}\bm{r}\big{\|}>0.$$

$$\mathbb{E}(X_{\mathbf{u}\mathbf{t}}-X_{\mathbf{u}\mathbf{s}}{)}^2\le \mathbb{E}(Y_{\mathbf{u}\mathbf{t}}-Y_{\mathbf{u}\mathbf{s}}{)}^2\text{for all }\mathbf{u},\mathbf{t},\mathbf{s};$$

$$\mathbb{E}(X_{\mathbf{u}\mathbf{t}}-X_{\mathbf{v}\mathbf{s}}{)}^2\ge \mathbb{E}(Y_{\mathbf{u}\mathbf{t}}-Y_{\mathbf{v}\mathbf{s}}{)}^2\text{for all }\mathbf{u}\ne \mathbf{v}\text{ and all }\mathbf{t},\mathbf{s}.$$

$$\mathbb{E}\underset{\mathbf{u}\in U}{\inf }\underset{\mathbf{t}\in T}{\sup }{X}_{\mathbf{u}\mathbf{t}}\le \mathbb{E}\underset{\mathbf{u}\in U}{\inf }\underset{\mathbf{t}\in T}{\sup }{Y}_{\mathbf{u}\mathbf{t}}.$$

$$\mathbb{P}\big{\{}f(X)-\mathbb{E}f(X)\geq t\big{\}}\leq e^{-t^{2}/(2L^{2})}.$$

$${\omega }^2(\mathcal{D}\cap S^{n-1})+{\omega }^2({\mathcal{D}}^{\circ }\cap S^{n-1})\le n.$$

$$F(\mathbf{\Psi })=\underset{\mathbf{t}\in {\Omega }_1}{\min }\underset{\mathbf{u}\in {\Omega }_2}{\max }⟨\mathbf{\Psi }\mathbf{u},\mathbf{t}⟩$$

$$\sqrt{m}<\sqrt{\omega^{2}\big{(}\mathcal{D}_{s}\cap S^{n-1}\big{)}+\omega^{2}\big{(}\mathcal{D}_{c}\cap S^{m-1}\big{)}}-t,$$

$$\min_{\|\bm{r}\|=1}\min_{\bm{s}\in\mathcal{D}_{s}^{\circ}\times\mathcal{D}_{c}^{\circ}}\big{\|}\bm{s}-\bm{A}^{*}\bm{r}\big{\|}>0$$

$$\underset{\Vert \mathbf{r}\Vert =1}{\min }\underset{\mathbf{s}\in {\mathcal{D}}_s^{\circ }\times {\mathcal{D}}_c^{\circ }}{\min }\Vert \mathbf{s}-{\mathbf{A}}^{\ast }\mathbf{r}{\Vert }_2>0$$

$$⟺\underset{\Vert \mathbf{r}\Vert =1}{\min }\underset{\mathbf{s}\in {\mathcal{D}}_s^{\circ }\times {\mathcal{D}}_c^{\circ }}{\min }\Vert \mathbf{s}-{\mathbf{A}}^{\ast }\mathbf{r}{\Vert }_2^2>0$$

$$\displaystyle\hskip 2.0pt\Longleftrightarrow\quad\min_{\|\bm{r}\|=1}\min_{\bm{s}_{1}\in\mathcal{D}_{s}^{\circ}\atop\bm{s}_{2}\in\mathcal{D}_{c}^{\circ}}\Big{[}\big{\|}\bm{s}_{1}-\bm{\Psi}^{*}\bm{r}\big{\|}_{2}^{2}+\big{\|}\bm{s}_{2}-\bm{r}\big{\|}_{2}^{2}\Big{]}>0.$$

$$\displaystyle\min_{\bm{r}\in\mathcal{D}_{c}^{\circ}\cap S^{m-1}}\min_{\bm{s}_{1}\in\mathcal{D}_{s}^{\circ}\atop\bm{s}_{2}\in\mathcal{D}_{c}^{\circ}}\Big{[}\big{\|}\bm{s}_{1}-\bm{\Psi}^{*}\bm{r}\big{\|}_{2}^{2}+\big{\|}\bm{s}_{2}-\bm{r}\big{\|}_{2}^{2}\Big{]}>0$$

$$\displaystyle\hskip 35.0pt\Longleftrightarrow\quad\min_{\bm{r}\in\mathcal{D}_{c}^{\circ}\cap S^{m-1}}\min_{\bm{s}_{1}\in\mathcal{D}_{s}^{\circ}}\big{\|}\bm{s}_{1}-\bm{\Psi}^{*}\bm{r}\big{\|}_{2}^{2}>0$$

$$\displaystyle\hskip 35.0pt\Longleftrightarrow\quad\min_{\bm{r}\in\mathcal{D}_{c}^{\circ}\cap S^{m-1}}\min_{\bm{s}_{1}\in\mathcal{D}_{s}^{\circ}}\big{\|}\bm{s}_{1}-\bm{\Psi}^{*}\bm{r}\big{\|}_{2}>0.$$

$$\underset{{\mathbf{s}}_1\in {\mathcal{D}}_s^{\circ }}{\min }\Vert {\mathbf{s}}_1-{\mathbf{\Psi }}^{\ast }\mathbf{r}{\Vert }_2$$

Full text

I Introduction

Corrupted sensing aims to recover a structured signal from a small number of corrupted measurements

[TABLE]

where $\bm{\Psi}\in\mathbb{R}^{m\times n}$ is the sensing measurement matrix which is assumed to have i.i.d. standard Gaussian entries in this paper, $\bm{x}^{\star}\in\mathbb{R}^{n}$ is the unknown signal, and $\bm{v}^{\star}\in\mathbb{R}^{m}$ is an unknown corruption. The goal is to estimate $\bm{x}^{\star}$ and $\bm{v}^{\star}$ from $\bm{y}$ and $\bm{\Psi}$.

This problem is encountered in many practical applications, such as face recognition [2], subspace clustering[3], network data analysis [4], and so on. Theoretical guarantees for this problem include sparse signal recovery from sparse corruption [5, 6, 7, 8, 9, 10, 11] and structured signal recovery from structured corruption [1, 12, 13].

To make the recovery possible, we will assume that both $\bm{x}$ and $\bm{v}$ have some structures which are promoted by the convex functions $f(\cdot)$ and $g(\cdot)$ respectively. When prior information about $f(\bm{x}^{\star})$ or $g(\bm{v}^{\star})$ is available, it is natural to consider the following program to recover the signal and corruption:

[TABLE]

or

[TABLE]

In [1], Foygel and Mackey provided conditions under which convex program (2) or (3) succeeds with high probability. Numerical experiments in [1] also suggested that there is a sharp phase transition when (2) or (3) is used to solve the corrupted sensing problem. However, little work has devoted to determining the threshold below which (2) or (3) fails with high probability. Therefore, theoretical understanding of the phase transition for program (2) and (3) is far from satisfactory.

In this paper, we present a theoretical analysis for the phase transition of (2) or (3). In particular, we figure out the exact position of phase transition, and demonstrate that the phase transition occurs in a relatively narrow region.

II Preliminaries

In this section, we present some preliminaries which will be used in our analysis.

Our result involves two important concepts: the Gaussian width and the tangent cone. Given a subset $T$ in $\mathbb{R}^{n}$, the Gaussian width is defined by

[TABLE]

We also define two tangent cones corresponding to signal and corruption respectively. The tangent cone of $f(\cdot)$ at the true signal $\bm{x}^{\star}$ is defined as

[TABLE]

Similarly, the tangent cone of $g(\cdot)$ at the true corruption $\bm{v}^{\star}$ is given by

[TABLE]

III Main results

In this section, we state our main results with some discussions.

Theorem 1** (Failure of convex program (2) or (3)).**

Consider convex program (2) or (3). Assume that both tangent cones $\mathcal{D}_{s}$ and $\mathcal{D}_{c}$ are closed. For any $t\geq 0$, if the measurement number $m$ satisfies

[TABLE]

*then the constrained convex program (2) or (3) fails with probability at least $1-\exp(-t^{2}/2)$, where $S^{n-1}$ and $S^{m-1}$ are the unit sphere of $\mathbb{R}^{n}$ and $\mathbb{R}^{m}$ respectively. *

Proof.

See Appendix A. ∎

Remark 1** (Phase transition of corrupted sensing).**

Recall Theorem $1$ and Remark $2$ in [1], which stated that 111The authors believe that the small additive constants are artifacts of the proof technique. 222The original result is stated in terms of Gaussian complexity $\gamma(\mathcal{D}_{s}\cap B^{n})$, difined as \gamma^{2}(\mathcal{D}_{s}\cap B^{n})=\mathbb{E}\big{(}\sup_{t\in\mathcal{D}_{s}\cap B^{n}}\left<g,t\right>\big{)}^{2}, where $B^{n}$ denotes the $\ell_{2}$ unit ball in $\mathbb{R}^{n}$. However, as the author stated, the Gaussian complexity $\gamma(\mathcal{D}_{s}\cap B^{n})$ is only very slightly larger than $\omega(\mathcal{D}_{s}\cap S^{n-1})$. when

[TABLE]

the constrained convex program (2) or (3) succeeds with probability at least $1-\exp(-t^{2}/2)$. This, together with our result Theorem 1, demonstrate that the phase transition of corrupted sensing occurs around

[TABLE]

and the width of phase transition area is about

[TABLE]

where $C$ is an absolute constant.

Remark 2**.**

Our result also agrees with the result of Amelunxen el al. [14]. Indeed, by Proposition 10.2 and Proposition 3.1 (9) in [14], we have

[TABLE]

where $\delta(\mathcal{D})$ denotes the statistical dimension of a convex cone $\mathcal{D}$.

Remark 3**.**

In [14], Amelunxen et al. considered the phase transition of the following demixing problem:

[TABLE]

where $\bm{x}$, $\bm{y}\in\mathbb{R}^{n}$ are unknown signals and $\bm{U}\in\mathbb{R}^{n\times n}$ is a random orthogonal matrix. This model is different from ours since we have random Gaussian measurement matrix with $m\ll n$.

Remark 4**.**

In [15], Oymak and Tropp considered the phase transition of the following demixing model:

[TABLE]

where $\bm{x}_{0}$, $\bm{x}_{1}\in\mathbb{R}^{n}$ are two signals and $\bm{\Psi}_{0}$, $\bm{\Psi}_{1}\in\mathbb{R}^{m\times n}$ are some random transformation matrices. This model is also different from ours since $\bm{\Psi}_{1}$ is a deterministic matrix in our case. This makes the problem more difficult to analyze.

IV Simulation Results

In this section, we employ a numerical experiment to verify our theoretical guarantees (Theorem 1). In the experiment, both signal and corruption are designed to be sparse vectors. We use CVX [16] [17] to solve the convex program (2) or (3).

In the experiment, we assume that the prior information of $f(\bm{x}^{\star})$ is known exactly, and solve program (3). The experiment settings are as follows: the ambient dimension $n$ is set to $128$, the measurement number $m=n=128$, the sparsity level of signal changes from $1$ to $n$ with step $1$, and the same for corruption. For every sparsity level of signal and corruption, we run and solve (3) $20$ times. We declare success if the solution to (3), denoted by $(\hat{\bm{x}},\hat{\bm{v}})$, satisfies $\|\hat{\bm{x}}-\bm{x}^{\star}\|_{2}\leq 10^{-3}$. Then we get the empirical probability of successful recovery. At last, we plot the theoretical curve predicted by Theorem 1.

Our numerical experiment result is shown in Fig. 1. We can see that the theoretical threshold given by Theorem 1 is closely matched with the empirical phase transition. It means that our theory can give a reliable prediction of the phase transition curve.

V Conclusion

This paper studied the problem of phase transition when we use convex program to solve corrupted sensing problem. Our results, together with previous work [1], gave the exact location of phase transition and the size of transition region. Simulations were provided to verify the correctness of our results. Our ongoing work is to establish a general framework to analyze the phase transition of various convex programs with noise-free or noisy data.

Appendix A Proof of Main Results

In this section, we present proof for our main result (Theorem 1). First, we will establish a sufficient condition under which convex program (2) or (3) fails, then some necessary tools are introduced, and at last, we give the proof for Theorem 1.

A-A Sufficient Condition for failure

In this subsection, we establish an easy-to-handle sufficient condition under which program (2) or (3) fails.

Lemma 1**.**

Let $\mathcal{D}_{s}$ and $\mathcal{D}_{c}$ denote the signal and the corruption tangent cones defined in (4) and (5) respectively. Then a sufficient condition under which constrained convex program (2) or (3) fails is

[TABLE]

In other words, the subset $\mathcal{D}_{s}\times\mathcal{D}_{c}\cap S^{n+m-1}$ intersects the null space of matrix $\begin{bmatrix}\bm{\Psi}&\bm{I}\end{bmatrix}$.

Proof.

Lemma 1 is a generalization of Proposition 2.1 of [18]. The proof is similar, and hence is omitted. ∎

Although Lemma 1 gives a sufficient condition for failure, it is difficult to check when (6) holds. The following lemma can overcome this drawback.

Lemma 2** (Sufficient condition for failure, Proposition 3.8, [15]).**

Under the condition of Lemma 1, if both $\mathcal{D}_{s}$ and $\mathcal{D}_{c}$ are closed, a sufficient condition for (6) to hold is

[TABLE]

where $(\mathcal{D}_{s}\times\mathcal{D}_{c})^{\circ}$ denotes the polar cone of $\mathcal{D}_{s}\times\mathcal{D}_{c}$, $\bm{A}=\begin{bmatrix}\bm{\Psi}&\bm{I}\end{bmatrix}$, and $\bm{I}$ denotes the identity matrix.

Remark 5**.**

One can easily check that

[TABLE]

Thus, the sufficient condition under which convex program (2) or (3) fails can be rewritten as

[TABLE]

In the following parts, we will prove that (8) holds with high probability when the condition of Theorem 1 is satisfied. Before this, let’s state some tools that will be used in our proof.

A-B Other Useful Tools

Lemma 3** (Gordon’s inequality, Theorem 3.16, [19]).**

Let $(X_{\bm{u}\bm{t}})_{\bm{u}\in U,\bm{t}\in T}$ and $(Y_{\bm{u}\bm{t}})_{\bm{u}\in U,\bm{t}\in T}$ be two Gaussian processes indexed by pairs of points $(\bm{u},\bm{t})$ in a product set $U\times T$. Assume that

[TABLE]

[TABLE]

Then we have

[TABLE]

Lemma 4** (Concentration of measure, Theorem 5.6, [20]).**

Let $X=(X_{1},\dots,X_{n})$ be a vector of $n$ independent standard normal random variables. Let $f$ : $\mathbb{R}^{n}\rightarrow\mathbb{R}$ denotes an L-Lipschitz function. Then, for all $t\geq 0$,

[TABLE]

Lemma 5** (Lemma 3.7, [18]).**

Let $\mathcal{D}\subset\mathbb{R}^{n}$ be a non-empty closed, convex cone. Then we have that

[TABLE]

Lemma 6**.**

Let $\Omega_{1}$ and $\Omega_{2}$ be subsets of $S^{m-1}$ and $S^{n-1}$ respectively. Then the function

[TABLE]

is a 1-Lipschitz function, where $\bm{\Psi}$ is the same as in (1).

Proof.

See Appendix B. ∎

A-C Proof of Main Results

According to Remark 5, we only need to prove that when

[TABLE]

the following event

[TABLE]

holds with probability at least $1-e^{-t^{2}/2}$. Moreover, a simple calculation verifies that this inequality is equivalent to

[TABLE]

Now, we will consider two cases for $\bm{r}$:

Case I: $\bm{r}\in\mathcal{D}_{c}^{\circ}\cap S^{m-1}$. In this case, when we minimize over $\bm{s}_{2}$, the second term \big{\|}\bm{s}_{2}-\bm{r}\big{\|}_{2}^{2} will be zero. Thus, the above inequality (A-C) is equivalent to

[TABLE]

For our purpose, we need to lower bound the left side of (A-C). Note that for any fixed $\bm{r}\in\mathcal{D}_{c}^{\circ}\cap S^{m-1}$, we have

[TABLE]

The first equality is due to the definition of $\ell_{2}$-norm. The first inequality is because of the minimax inequality. The second equality comes from the linear property of inner product. The third equality uses the fact that $\max_{\bm{s}\in\mathcal{D}_{s}^{\circ}}\left<\bm{u},\bm{s}\right>=0$ when $\bm{u}\in\mathcal{D}_{s}$, otherwise it equals $\infty$. The last equality can be derived by a simple transformation. As the above inequality holds for any $\bm{r}\in\mathcal{D}_{c}^{\circ}\cap S^{m-1}$, we have

[TABLE]

It remains to bound the right side. To this end, we will first use Gordon’s inequality (Lemma 3) to derive a lower bound for the expectation, and then concentration of measure (Lemma 4) to obtain the desired result. Let $X_{\bm{r}\bm{u}}:=\left<\bm{\Psi}\bm{u},\bm{r}\right>$ and $Y_{\bm{r}\bm{u}}:=\left<\bm{g},\bm{r}\right>+\left<\bm{h},\bm{u}\right>$ be two Gaussian processes, where $\bm{g}\sim N(\bm{0},\bm{I}_{m\times m})$ and $\bm{h}\sim N(\bm{0},\bm{I}_{n\times n})$ are independent standard Gaussian random vectors. It can be easily checked that the increments satisfy

[TABLE]

[TABLE]

Therefore, Gordon’s inequality (Lemma 3) gives us:

[TABLE]

Since $\bm{g}$ is a symmetric random vector, we have

[TABLE]

Substituting this into (A-C), we get

[TABLE]

As $\mathcal{D}_{c}$ is a closed convex cone, by Lemma 5, we know that

[TABLE]

which implies

[TABLE]

Substituting this into (13), we get the following result:

[TABLE]

In the last inequality, we have used the assumption that $\omega^{2}(\mathcal{D}_{s}\cap S^{n-1})+\omega^{2}(\mathcal{D}_{c}\cap S^{m-1})>m$.

Next, Lemma 6 confirms that the following function

[TABLE]

is a $1$-Lipschitz function. Thus, concentration of measure (Lemma 4) gives us that for any $t\geq 0$,

[TABLE]

Putting the above inequality and (A-C), (11), (A-C), (A-C) together, we eventually get that when

[TABLE]

we have

[TABLE]

Case II: $\bm{r}\notin\mathcal{D}_{c}^{\circ}\cap S^{m-1}$. In this case, it is clear that no matter what $\bm{r}$ and $\bm{s}_{2}$ takes value, it is always holds that

[TABLE]

Thus,

[TABLE]

which, by (A-C) and (A-C), implies that

[TABLE]

Union bound. Combining case I and case II and taking a union bound, we have

[TABLE]

provided

[TABLE]

By Lemma 1 and Lemma 2, it means that when

[TABLE]

the convex program (2) or (3) fails with probability at least $1-\exp(-t^{2}/2)$. This completes the proof.

Appendix B Proof of Lemma 6

To prove Lemma 6, we only need to show that for any $\bm{C},\bm{D}\in\mathbb{R}^{m\times n}$

[TABLE]

For any fixed $\bm{t}\in\Omega_{1}$, let

[TABLE]

And we have

[TABLE]

Then, let

[TABLE]

and we have

[TABLE]

Similarly,

[TABLE]

Therefore,

[TABLE]

The same argument gives

[TABLE]

Thus, combining (B) and (16), we get

[TABLE]

The conclusion follows immediately.