Sharp Guarantees for Solving Random Equations with One-Bit Information

TL;DR

This paper provides sharp theoretical guarantees for convex optimization methods in recovering signals from one-bit measurements in high-dimensional settings, applicable to various loss functions and validated by simulations.

Contribution

It offers a unified analysis predicting the performance of a broad class of convex estimators for one-bit compressed sensing, including new bounds for several loss functions.

Findings

Performance predictions match simulations in high dimensions

Bounds apply to multiple convex loss functions

Results hold for Gaussian measurement vectors

Abstract

We study the performance of a wide class of convex optimization-based estimators for recovering a signal from corrupted one-bit measurements in high-dimensions. Our general result predicts sharply the performance of such estimators in the linear asymptotic regime when the measurement vectors have entries IID Gaussian. This includes, as a special case, the previously studied least-squares estimator and various novel results for other popular estimators such as least-absolute deviations, hinge-loss and logistic-loss. Importantly, we exploit the fact that our analysis holds for generic convex loss functions to prove a bound on the best achievable performance across the entire class of estimators. Numerical simulations corroborate our theoretical findings and suggest they are accurate even for relatively small problem dimensions.

Equations

$${\hat{\mathbf{x}}}_{\ell }:=\arg \underset{\mathbf{x}}{\min }\sum _{i=1}^m\ell (y_i{\mathbf{a}}_i^T\mathbf{x}).$$

$$\sum _{i=1}^m\ell (y_i{\mathbf{a}}_i^T\mathbf{x})+r{\Vert \mathbf{x}\Vert }_2^2.$$

$$\hat{\mathbf{x}}:=\arg \underset{\mathbf{x}}{\min }\sum _{i=1}^m\tilde{\ell }(y_i-{\mathbf{a}}_i^T\mathbf{x})+rR(\mathbf{x})$$

$$y_i={\mathrm{BSC}}_{\epsilon }({\overline{y}}_i):=\{\begin{array}{ll}\mathrm{sign}({\mathbf{a}}_i^T{\mathbf{x}}_0)& ,\text{w.p.}1-\epsilon ,\\ -\mathrm{sign}({\mathbf{a}}_i^T{\mathbf{x}}_0)& ,\text{w.p.}\epsilon .\end{array}$$

$${\hat{\mathbf{x}}}_{\ell }\in \arg \underset{\mathbf{x}}{\min }\frac{1}{m}\sum _{i=1}^m\ell (y_i{\mathbf{a}}_i^T\mathbf{x}),$$

$$\mathrm{corr}\left({\hat{\mathbf{x}}}_{\ell };{\mathbf{x}}_0\right):=\frac{|⟨{\hat{\mathbf{x}}}_{\ell },{\mathbf{x}}_0⟩|}{\Vert {\hat{\mathbf{x}}}_{\ell }{\Vert }_2\Vert {\mathbf{x}}_0{\Vert }_2}\in [0,1].$$

$$y_i=\{\begin{array}{ll}1& ,\text{w.p.}f({\mathbf{a}}_i^T{\mathbf{x}}_0),\\ -1& ,\text{w.p.}1-f({\mathbf{a}}_i^T{\mathbf{x}}_0).\end{array}$$

$${\hat{\mathbf{x}}}_{\ell }=\arg \underset{\mathbf{x}}{\min }\sum _{i=1}^m\ell (y_i{\mathbf{a}}_i^T\mathbf{x})+r{\Vert \mathbf{x}\Vert }_2^2,$$

$${\mathcal{M}}_{\ell }\left(x;\lambda \right):=\underset{v}{\min }\frac{1}{2\lambda }(x-v{)}^2+\ell (v),$$

$${\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(x;\lambda \right)$$

$${\mathcal{M}}_{\ell ,2}^{{}^{\prime }}\left(x;\lambda \right)$$

$${\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(x;\lambda \right)$$

$${\mathcal{M}}_{\ell ,2}^{{}^{\prime }}\left(x;\lambda \right)$$

$${\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(x;\lambda \right)$$

$${\mathcal{M}}_{\ell ,2}^{{}^{\prime }}\left(x;\lambda \right)$$

$$G,S\stackrel{\text{iid}}{\sim }\mathcal{N}(0,1)\text{and}Y={\mathrm{BSC}}_{\epsilon }(\mathrm{sign}(S)),$$

$$\mathbb{E}\left[YS\cdot {\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(\alpha G+\mu SY;\lambda \right)\right]$$

$${\lambda }^2\delta \mathbb{E}\left[{\left({\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(\alpha G+\mu SY;\lambda \right)\right)}^2\right]$$

$$\lambda \delta \mathbb{E}\left[G\cdot {\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(\alpha G+\mu SY;\lambda \right)\right]$$

$$\mathbb{E}\left[YS\cdot {\ell }^{{}^{\prime }}\left({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)\right)\right]$$

$${\lambda }^2\delta \mathbb{E}\left[{\left({\ell }^{{}^{\prime }}\left({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)\right)\right)}^2\right]$$

$$\lambda \delta \mathbb{E}\left[G\cdot {\ell }^{{}^{\prime }}\left({\mathrm{prox}}_{\ell }\left(\alpha G+\mu SY;\lambda \right)\right)\right]$$

$$1$$

$$\underset{n\to \infty }{\lim }\mathrm{corr}\left({\hat{\mathbf{x}}}_{\ell };{\mathbf{x}}_0\right)=\sqrt{\frac{1}{1+(\alpha /\mu {)}^2}}.$$

$$\displaystyle\lim_{n\rightarrow\infty}\Big{\|}\widehat{\mathbf{x}}_{\ell}-\mu\cdot\frac{\mathbf{x}_{0}}{\|\mathbf{x}_{0}\|_{2}}\Big{\|}_{2}^{2}=\alpha^{2}.$$

$$\exists {\mathbf{x}}_s\in {\mathbb{R}}^p\text{such that}{y}_i{\mathbf{a}}_i^T{\mathbf{x}}_s\ge 1,\forall i\in [m].$$

$$\delta \le {\delta }_{\epsilon }^{\star }:={\left(\underset{c\in \mathbb{R}}{\min }\mathbb{E}\left[{\left(G+cSY\right)}_{-}^2\right]\right)}^{-1},$$

$$\delta =\frac{\mathbb{E}\left[{\left({\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(\alpha G+\mu SY;\lambda \right)\right)}^2\right]}{{\left(\mathbb{E}\left[G\cdot {\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(\alpha G+\mu SY;\lambda \right)\right]\right)}^2}\ge 1.$$

$$\displaystyle\sigma_{\ell}^{2}\,\mathcal{I}\big{(}\sigma_{\ell}\,G+SY)\geq\frac{1}{\delta}.$$

$$1=\lambda \delta \mathbb{E}[{\mathcal{M}}_{\ell ,1}^{{}^{\prime \prime }}\left(\alpha G+\mu SY;\lambda \right)],$$

$${\alpha }^2=\frac{1}{\delta }\cdot \frac{\mathbb{E}\left[{\left({\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(\alpha G+\mu SY;\lambda \right)\right)}^2\right]}{{\left(\mathbb{E}[{\mathcal{M}}_{\ell ,1}^{{}^{\prime \prime }}\left(\alpha G+\mu SY;\lambda \right)]\right)}^2}.$$

$$Z:=\mu SY\text{and}W:=\alpha G+Z.$$

$$p_W^{{}^{\prime }}(w)=\int {\varphi }_{\alpha }^{{}^{\prime }}(u)p_Z(w-u)\mathrm{d}u,$$

$$\mathbb{E}\left[{\left({\mathcal{M}}_{\ell ,1}^{{}^{\prime }}\left(W;\lambda \right)\right)}^2\right]\cdot \mathcal{I}(W)$$

Full text

I Introduction

I-A Motivation

Classical statistical signal-processing theory studies estimation problems in which the number of unknown parameters $n$ is small compared to the number of observations $m$. In contrast, modern inference problems are typically high-dimensional, that is $n$ can be of the same order as $m$. Examples are abundant in a wide range of signal-processing applications such as medical imaging, wireless communications, recommendation systems and so on. Classical tools and theories are not applicable in these modern inference problems. As such, over the last two decades or so, the study of high-dimensional estimation problems has received significant attention. Despite the remarkable progress in many directions, several important questions remain to be explored.

This paper studies the fundamental problem of recovering an unknown signal from (possibly corrupted) one-bit measurements in high-dimensions. We focus on a rather rich class of convex optimization-based estimators that includes, for example, least-squares (LS), least-absolute deviations (LAD), logistic regression and hinge-loss as special cases. For such estimators and Gaussian measurement vectors, we compute their asymptotic performance in the high-dimensional linear regime in which $m,n\rightarrow+\infty$ and $m/n\rightarrow\delta\in(1,+\infty)$. Importantly, our results are sharp. In contrast to existing related results which are order-wise (i.e., they involve unknown or loose constants) this allows us to accurately compare the relative performance of different methods (e.g., LS vs LAD). It is worth mentioning that while our predictions are asymptotic, our numerical illustrations suggest that they are valid for dimensions $m$ and $n$ that are as small as a few hundreds.

I-B Contributions

Our goal is to recover $\mathbf{x}_{0}\in\mathbb{R}^{n}$ from measurements $y_{i}=\mathrm{sign}(\mathbf{a}_{i}^{T}\mathbf{x}_{0}),~{}i=1,\ldots,m$, where $\mathbf{a}_{i}\in\mathbb{R}^{n}$ have entries iid Gaussian. The results account for possible corruptions by allowing each measurement $y_{i}$ to be sign-flipped with constant probability $\varepsilon\in[0,1/2]$ (see Section II for details). We study the asymptotic performance of estimators $\hat{\mathbf{x}}_{\ell}$ that are solutions to the following optimization problem for some convex loss function $\ell(\cdot)$.

[TABLE]

When $m,n\rightarrow+\infty$ and $m/n\rightarrow\delta>\delta_{\varepsilon}^{\star}$, we show that the correlation of $\widehat{\mathbf{x}}_{\ell}$ to the true vector $\mathbf{x}_{0}$ is sharply predicted by $\sqrt{\frac{1}{1+({\alpha}/{\mu})^{2}}}$ where the parameters $\alpha$ and $\mu$ are the solutions to a system of three non-linear equations in three unknowns. We find that the system of equations (and thus, the value of $\alpha/\mu$) depends on the loss function $\ell(\cdot)$ through its Moreau envelope function. We prove that $\delta_{\varepsilon}^{\star}>1$ is necessary for the equations to have a bounded solution, but, in general, the value of the threshold $\delta_{\varepsilon}^{\star}$ depends both on the noise level $\varepsilon$ and on the loss function. For the general case where $\varepsilon\in[0,1/2]$, we propose a method to find the upper bound on correlation which corresponds to the maximum value of correlation that can be achieved for any convex loss function

Despite the fact that it is not the main focus of this paper, we remark that our results hold for the general case where measurements $y_{i}$ are determined according to an arbitrary function $f:\mathbb{R}\rightarrow[0,1]$ (see (6)). Moreover as our analysis in Appendix A-B shows, the system of equations introduced in Section III-B can be extended to address regularized loss functions i.e.,

[TABLE]

We specialize our general result to specific loss functions such as LS, LAD and hinge-loss. This allows us to numerically compare the performance of these popular estimators by simply evaluating the corresponding theoretical predictions. Our numerical illustrations corroborate our theoretical predictions. For LS, our equations can be solved in closed form and recover the result of [29] (see Section I-C). For the hinge-loss, we show that $\delta_{\varepsilon}^{\star}$ is a decreasing function of $\varepsilon$ that approaches $+\infty$ in the noiseless case and $2$ when $\varepsilon=1/2$. We believe that our work opens the possibility for addressing several important open questions, such as finding the optimal choice of the loss function in (1) and the value of $\delta_{\varepsilon}^{\star}$ for general loss functions.

I-C Prior work

As mentioned, over the past two decades there has been a very long list of works that derive statistical guarantees for high-dimensional estimation problems. In particular, many of these are concerned with convex optimization-based inference methods. Our work is most closely related to the following two lines of research.

(a) Sharp asymptotic predictions for noisy linear measurements. Most of the results in the literature of high-dimensional statistics are order-wise in nature. Sharp asymptotic predictions have only recently appeared in the literature for the case of noisy linear measurements with Gaussian measurement vectors. There are by now three different approaches that have been used (to different extent each) towards asymptotic analysis of convex regularized estimators: (a) the one that is based on the approximate message passing (AMP) algorithm and its state-evolution analysis; [11, 12, 2, 3, 10] (b) the one that is based on Gaussian process (GP) inequalities, specifically the convex Gaussian min-max Theorem (CGMT); [26, 8, 27, 20, 31, 30] (c) and the “leave-one-out” approach [19, 13]. The three approaches are quite different to each other and each comes with its unique distinguishing features and disadvantages. A detailed comparison is beyond our scope. In this paper, we follow the GP approach and build upon the CGMT. Since concerned with linear measurements, these previous works consider estimators that solve minimization problems of the form

[TABLE]

Specifically, the loss function $\widetilde{\ell}(\cdot)$ penalizes the residual. In this paper, we extend the applicability of the CGMT to optimization problems in the form of (1). For our case of signed measurements, (1) is more general than (2). To see this, note that for $y_{i}\in\pm 1$ and popular symmetric loss functions $\widetilde{\ell}(t)=\widetilde{\ell}(-t)$ (e.g., LS, LAD), (1) results in (2) by choosing $\ell(t)=\widetilde{\ell}(t-1)$ in the former. Moreover, (1) includes several other popular loss functions such as the logistic loss and the hinge-loss which cannot be expressed by (2).

(b) One-bit compressed sensing. Our works naturally relates to the literature of one-bit compressed sensing (CS) [5]. The vast majority of performance guarantees for one-bit CS are order-wise in nature, e.g., [18, 22, 21, 23]. To the best of our knowledge, the only existing sharp results are presented in [29] for Gaussian measurement vectors. Specifically, the paper [29] derives the asymptotic performance of regularized LS for generalized nonlinear measurements, which include signed measurements as a special case. Our work can be seen as a direct extension of [29] to loss functions beyond least-squares, such as the hinge-loss. In fact, the result of [29] for our setting is a direct corollary of our main theorem (see Section IV-A). As in [29], our proof technique is based on the CGMT.

There are few works that consider general convex loss functions for estimating a signal from noisy measurements in high dimensions. In [33], the general estimator $\ell(\langle\mathbf{a}_{i},\mathbf{x}\rangle,y_{i})$ for estimating a structured signal in the non-asymptotic case has been studied. However it is assumed that the loss function satisfies some conditions including restricted strong convexity, continuously differentiability in the first argument and derivative of loss function being Lipschitz-continuous with respect to the second argument. The author furthermore derives some sufficient conditions for the loss function ensuring the restricted strong convexity condition. Our result in Theorem III.2 for achieving the best performance across all loss functions is comparable to [34, Theorem 1] in which they have also proposed a method for deriving optimal loss function and measuring its performance. However their results hold for measurements of the form $y_{i}=\mathbf{a}_{i}^{T}\mathbf{x}_{0}+\varepsilon_{i}$, where $\{\varepsilon_{i}\}_{i=1}^{m}$ are random errors independent of $\mathbf{a}_{i}$’s.

Finally, our paper is closely related to [7, 28], in which the authors study the high-dimensional performance of maximum-likelihood (ML) estimation for the logistic model. The ML estimator is a special case of (1) but their measurement model differs from the one considered in this paper. Also, their analysis is based on the AMP. While this paper was being prepared, we became aware of [25], in which the authors extend the results of [28] to regularized ML by using the CGMT. However we present results for general loss functions and a different measurement model.

I-D Organization and notation

The rest of the paper is organized as follows. Section II formally introduces the problem that this paper is concerned with. We present our main results Theorems III.1 and III.2 in Section III, where we also discuss some of their implications. In Section IV, we specialize the general result of Theorem III.1 to the LS, LAD and hinge-loss estimators. We also present numerical simulations to validate our theoretical predictions. We conclude in Section V with several possible directions for future research. A proof sketch of Theorem III.1 is provided in Appendix A.

The symbols $\mathbb{P}\left(\cdot\right)$ and $\mathbb{E}\left[\cdot\right]$ denote the probability of an event and the expectation of a random variable, respectively. We use boldface notation for vectors. $\|\mathbf{v}\|_{2}$ denotes the Euclidean norm of a vector $\mathbf{v}$. We write $i\in[m]$ for $i=1,2,\ldots,m$. When writing $x_{*}=\arg\min_{x}f(x),$ we let the operator $\arg\min$ return any one of the possible minimizers of $f$. For all $x\in\mathbb{R}$, Gaussian $Q$-function at $x$ is defined as $Q(x)=\int_{s=x}^{\infty}\frac{1}{\sqrt{2\pi}}e^{s^{2}/2}ds.$ For a random variable $H$ with density $p_{H}(h)$ that has a derivative $p_{H}^{{}^{\prime}}(h)$ for all $h\in\mathbb{R}$, denote its score function $\xi_{h}:=\frac{\partial}{\partial h}{\log p(h)}=\frac{p_{H}^{{}^{\prime}}(h)}{p_{H}(h)}$. Fisher information of $H$ (e.g., [1, Sec. 2]) is defined as $\mathcal{I}(H):=\mathbb{E}[\,(\xi_{H})^{2}\,].$

II Problem statement

The goal is to recover the unknown vector $\mathbf{x}_{0}\in\mathbb{R}^{n}$ from $m$ noisy signed measurements $y_{i}$. Let $\overline{y}_{i},~{}i\in[m]$ denote the noiseless signed measurements $\overline{y}_{i}=\mathrm{sign}(\mathbf{a}_{i}^{T}\mathbf{x}_{0}),~{}i\in[m]$, where $\mathbf{a}_{i}\in\mathbb{R}^{n}$ are the measurement vectors. We assume the following noise model in which each measurement is corrupted (i.e., sign flipped) with some constant probability $\varepsilon\in[0,1/2]$:

[TABLE]

We remark that all our results remain valid in the case of (potentially) adversarial noise in which $\varepsilon\,m$ number of noiseless measurements $\overline{y}_{i}$ are flipped. Nevertheless, for the rest of the paper, we focus on the measurement model in (3). This paper studies the recovery performance of estimates $\widehat{\mathbf{x}}_{\ell}$ of $\mathbf{x}_{0}$ that are obtained by solving the following convex optimization problem:

[TABLE]

Here, $\ell:\mathbb{R}\rightarrow\mathbb{R}$ is a convex loss function and the subscript $\ell$ on the estimate $\widehat{\mathbf{x}}_{\ell}$ emphasizes its dependence on the choice of the loss function. Different choices for $\ell(.)$ lead to popular specific estimators. For example, these include the following:

• •

Least-squares (LS): $\ell(t)=\frac{1}{2}(t-1)^{2}$,

• •

Least-absolute deviations (LAD): $\ell(t)=|t-1|$,

• •

Logistic maximum-likelihood: $\ell(t)=\log(1+e^{-t})$,

• •

Ada-boost: $\ell(t)=e^{-t}$,

• •

Hinge-loss: $\ell(t)=\max\{1-t\,,\,0\}$.

Since we only observe sign-information, any information about the magnitude $\|\mathbf{x}_{0}\|_{2}$ of the signal $\mathbf{x}_{0}$ is lost. Thus, we can only hope to obtain an accurate estimate of the direction of $\mathbf{x}_{0}$. We measure performance of the estimate $\widehat{\mathbf{x}}_{\ell}$ by its (absolute) correlation value to $\mathbf{x}_{0}$, i.e.,

[TABLE]

Obviously, we seek estimates that maximize correlation.

Although signed measurements obtained according to (3) are the main focus of this paper, we remark that all results are valid for the general case where measurements are determined according to :

[TABLE]

where $f:\mathbb{R}\rightarrow[0,1]$. It is straightforward to see that choosing $f(t)=\frac{1}{2}+\frac{1-2\varepsilon}{2}\mathrm{sign}(t)$ will give (3). Moreover as the analysis in Appendix A-B suggests, we can extend our results to the broader class of optimization problems where there is an additional term for the penalty on the magnitude of $\mathbf{x}$, i.e.,

[TABLE]

where $r\in\{\mathbb{R}^{+}\cup{0}\}$. However, the main body of this paper focuses on the case $r=0$.

Our main result characterizes the asymptotic value of ${\rm{corr}}\left(\,{\widehat{\mathbf{x}}_{\ell}}\,;\,{\mathbf{x}_{0}}\,\right)$ in the linear high-dimensional regime in which the problem dimensions $m$ and $n$ grow proportionally to infinity with ${m}/{n}\rightarrow\delta\in(1,\infty).$ All our results are valid under the assumption that the measurement vectors have entries IID Gaussian.

Assumption 1** (Gaussian measurement vectors).**

The vectors $\mathbf{a}_{i},~{}i\in[m]$ have entries IID standard normal $\mathcal{N}(0,1)$.

We make no further assumptions on the distribution of the true vector $\mathbf{x}_{0}$.

III Main results

III-A Moreau Envelopes

Before presenting our main result, we need a few definitions. We write

[TABLE]

for the Moreau envelope function of the loss $\ell:\mathbb{R}\rightarrow\mathbb{R}$ at $x$ with parameter ${\lambda}>0$. Note that the objective function in the minimization above is strongly convex. Thus, for all values of $x$ and ${\lambda}$, there exists a unique minimizer which we denote by $\mathrm{prox}_{{\ell}}\left({x};{{\lambda}}\right)$. This is known as the proximal operator of $\ell$ at $x$ with parameter ${\lambda}$. One of the important and useful properties of the Moreau envelope function is that it is continuously differentiable with respect to both $x$ and ${\lambda}$ [24]. We denote these derivatives as follows

[TABLE]

The following is a well-known result that is useful for our purposes.

Proposition III.1** (Derivatives of $\mathcal{M}_{\ell}$ [24]).**

For a function $\ell:\mathbb{R}\rightarrow\mathbb{R}$, and all $x\in\mathbb{R}$ and $\lambda>0$, the following properties are true:

[TABLE]

If in addition $\ell$ is differentiable and $\ell^{{}^{\prime}}$ denotes its derivative, then

[TABLE]

III-B A system of equations

It turns out, that the asymptotic performance of (4) depends on the loss function $\ell$ via its Moreau envelope. Specifically, define random variables $G,S$ and $Y$ as follows (recall the definition of ${\rm{BSC}}_{\varepsilon}$ in (3))

[TABLE]

and consider the following system of non-linear equations in three unknowns $(\mu,\alpha>0,{\lambda})$:

[TABLE]

The expectations above are with respect to the randomness of the random variables $G$, $S$ and $Y$. As we show shortly, the solution to these equations is tightly connected to the asymptotic behavior of the optimization in (4).

We remark that the equations are well defined even if the loss function $\ell$ is not differentiable. If $\ell$ is differentiable then, using Proposition III.1 the Equations (9) can be equivalently written as follows:

[TABLE]

Finally, if $\ell$ is two times differentiable then applying integration by parts in Equation (10c) results in the following reformulation of (9c):

[TABLE]

Note that the system of equations in (9) and (10) hold when there is no regularization i.e., $r=0$. For the general case, the system of equations can be extended to (45).

III-C Asymptotic prediction

We are now ready to state the main result of this paper.

Theorem III.1** (General loss function).**

Let Assumption 1 hold and fix some $\varepsilon\in[0,1/2]$ in (3). Assume $\delta>1$ such that the set of minimizers in (4) is bounded and the system of Equations (9) has a unique solution $(\mu,\alpha,{\lambda})$, such that $\mu\neq 0$. Let $\widehat{\mathbf{x}}_{\ell}$ be as in (4). Then, in the limit of $m,n\rightarrow+\infty$, $m/n\rightarrow\delta$, it holds with probability one that

[TABLE]

Moreover,

[TABLE]

Theorem III.1 holds for general loss functions. In Section IV we specialize the result to specific popular choices. We also present numerical simulations that confirm the validity of the predictions of Theorem III.1 (see Figures 3–5). Before that, in the following remarks we present a few notes on the conditions, interpretation and implications of the theorem. A proof outline is included in Appendix A.

Remark 1* (The role of $\mu$ and $\alpha$).*

The theorem evaluates the asymptotic performance of the estimator $\widehat{\mathbf{x}}_{\ell}$ for a convex loss function $\ell$ in (4). According to (12), the prediction for the limiting behavior of the correlation value is given in terms of \sigma_{\ell}:=\alpha\big{/}\mu, where $\mu$ and $\alpha$ are unique solutions of (9). The smaller the value of $\sigma_{\ell}$ is, the larger becomes the correlation value. Thus, the correlation value is fully determined by the ratio of the parameters $\alpha$ and $\mu$. Their individual role is clarified in (13). Specifically, according to (13), $\widehat{\mathbf{x}}_{\ell}$ is a biased estimate of the true $\mathbf{x}_{0}$ and $\mu$ represents exactly that bias term. In other words, solving (4) returns an estimator that is close to a $\mu$–scaled version of $\mathbf{x}_{0}$. When $\mathbf{x}_{0}$ and $\widehat{\mathbf{x}}_{\ell}$ are scaled appropriately, then the L2 squared norm of their difference converges to $\alpha^{2}$.

Remark 2* (On the existence of a solution to (4)).*

While $\delta>1$ is a necessary condition for the equations in (9) to have a solution, it is not sufficient in general. This depends on the specific choice of the loss function. For example, in Section IV-A, we show that for the squared loss $\ell(t)=(t-1)^{2}$, the equations have a unique solution iff $\delta>1$. On the other hand, for logistic regression and hinge-loss, it is argued in Remark 3 that there exists a threshold value $\delta^{\star}_{\varepsilon}:=\delta^{\star}(\varepsilon)>2$ such that the set of minimizers in (4) is unbounded if $\delta>\delta_{\varepsilon}$. Hence, the theorem does not hold for $\delta<\delta^{\star}_{\varepsilon}$. We conjecture that for these choices of loss, the equations (9) are solvable iff $\delta>\delta_{\varepsilon}$. Justifying this conjecture is an interesting direction for future work. More generally, we leave the study of sufficient and necessary conditions under which the equations (9) admit a unique solution to future work.

Remark 3* (Bounded minimizers).*

Theorem III.1 only holds in regimes for which the set of minimizers of (4) is bounded. As we show here, this is not always the case. Specifically, consider non-negative loss functions $\ell(t)\geq 0$ with the property $\lim_{t\rightarrow+\infty}\ell(t)=0$. For example, the hinge-loss, Ada-boost and logistic loss all satisfy this property. Now, we show that for such loss functions the set of minimizers is unbounded if $\delta<\delta^{\star}_{\varepsilon}$ for some appropriate $\delta^{\star}_{\varepsilon}>2$. First, note that the set of minimizers is unbounded if the following condition holds:

[TABLE]

Indeed, if (14) holds then $\mathbf{x}=c\cdot\mathbf{x}_{s}$ with $c\rightarrow+\infty$, attains zero cost in (4); thus, it is optimal and the set of minimizers is unbounded. To proceed, we rely on a recent result by Candes and Sur [7] who prove that (14) holds iff 111To be precise, [7] prove the statement for measurements $y_{i},~{}i\in[m]$ that follow a logistic model. Close inspection of their proof shows that this requirement can be relaxed by appropriately defining the random variable $Y$ in (15).

[TABLE]

where $G,S$ and $Y$ are random variables as in (8) and $(t)_{-}:=\min\{0,t\}$. It can be checked analytically that $\delta^{\star}(\varepsilon)$ is a decreasing function of $\varepsilon$ with $\delta^{\star}(0^{+})=+\infty$ and $\delta^{\star}(1/2)=2$. In Figure 1, we have numerically evaluated the threshold value $\delta^{\star}_{\varepsilon}$ as a function of the corruption level $\varepsilon$. For $\delta<\delta^{\star}_{\varepsilon}$, the set of minimizers of the (4) with logistic or hinge loss is unbounded.

Remark 4* (Why $\delta>1$).*

The theorem assumes that $\delta>1$ or equivalently $m>n$. Here, we show that this condition is necessary for the equations (9) to have a bounded solution, or equivalently for the minimizer $\widehat{\mathbf{x}}_{\ell}$ in (4) to be bounded. To see this, take squares in both sides of (9c) and divide by (9b), to find that

[TABLE]

The inequality follows by applying Cauchy-Schwarz and using the fact that $\mathbb{E}[G^{2}]=1$.

Remark 5* (Solving the equations).*

Evaluating the performance of $\widehat{\mathbf{x}}_{\ell}$ requires solving the system of non-linear equations in (4). We empirically observe (see also [30] for similar observation) that if a solution exists, then it can be efficiently found by the following fixed-point iteration method. Let $\mathbf{v}:=[\mu,\alpha,{\lambda}]^{T}$ and $\mathcal{F}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{3}$ be such that (4) is equivalent to $\mathbf{v}=\mathcal{F}(\mathbf{v})$. With this notation, initialize $\mathbf{v}=\mathbf{v}_{0}$ and for $k\geq 1$ repeat the iterations $\mathbf{v}_{k+1}=\mathcal{F}(\mathbf{v}_{k})$ until convergence.

III-D Performance bounds

In this section, we use the asymptotic prediction of Theorem III.1 to derive an upper bound on the correlation value ${\rm{corr}}\left(\,{\widehat{\mathbf{x}}_{\ell}}\,;\,{\mathbf{x}_{0}}\,\right)$ that holds for all choices of continuously differentiable loss functions. The following theorem is the key intermediate result in this direction. At a high-level, the proof of the theorem involves a careful algebraic manipulation of the system of equations (9), and, leveraging properties of the Moreau envelope function.

Theorem III.2** (Lower bound on $\sigma_{\ell}$).**

Let $\ell$ be continuously differentiable and let assumptions and notation of Theorem III.1 hold. Then,

[TABLE]

Proof.

First, applying Gaussian integration by parts in (9c) we find that :

[TABLE]

where $\mathcal{M}^{{}^{\prime\prime}}_{{\ell},1}\left({x};{{\lambda}}\right)$ denotes the second derivative of the Moreau-envelope function with respect to the first argument (which exists since $\ell$ is differentiable). Solving for $\lambda$ and substituting in (9b) gives :

[TABLE]

The rest of the proof follows steps similar to [10, Lem. 3.4] and [30, Rem. 5.3.3]. Call

[TABLE]

Note that for $\alpha\neq 0$, $W$ has a continuously differentiable function at every $w$; in fact,

[TABLE]

where $\phi_{\alpha}(u)=\frac{1}{\alpha\sqrt{2\pi}}e^{-\frac{u^{2}}{2\alpha^{2}}}$.

With this notation, by Cauchy-Schwartz inequality we have:

[TABLE]

where the last line follows by integration by parts. This, when combined with (17) shows that:

[TABLE]

Recall that $W=\alpha G+\mu SY$ and $G\sim\mathcal{N}(0,1)$. Now, since (e.g., [1, Eqn. 2.13])

[TABLE]

we have that

[TABLE]

where recall that

[TABLE]

Substituting this in (18) gives the desired inequality in the statement of the theorem. ∎

As it is illustrated in Figure 2, \sigma_{\ell}^{2}\,\mathcal{I}\big{(}\sigma_{\ell}\,G+SY) is an increasing function of $\sigma_{\ell}$. Therefore the minimum value of $\sigma_{\ell}$ can be derived from the inequality in (16), which corresponds to the optimum $\sigma_{\ell}$ that can be achieved for any continuously differentiable loss function. In the following two remarks, this result is extended to achieve the optimum values for correlation.

Remark 6* (Upper bound on correlation).*

Using (16) with $Y={\rm{BSC}}_{\epsilon}(\mathrm{sign}(S))$ and by numerically evaluating $\mathcal{I}(\sigma_{\ell}G+SY)$ based on $\sigma_{\ell}$ , we can find a numerical lower bound on $\sigma_{\ell}$ . In view of (12) this directly yields an upper bound on ${\rm{corr}}\left(\,{\widehat{\mathbf{x}}_{\ell}}\,;\,{\mathbf{x}_{0}}\,\right)$, i.e., the maximum correlation that can be achieved for any choice of (continuously differentiable) convex loss function. In Figures 3-5, the green lines show the upper bound on correlation for specific values of $\varepsilon$; see Section IV-D for details.

IV Special cases

In this section, we apply the general result of Theorem III.1 to specific popular choices of the loss function.

IV-A Least-squares

By choosing $\ell(t)=(t-1)^{2}$ in (4), we obtain the standard least squares estimate. To see this, note that since $y_{i}=\pm 1$, it holds for all $i$ that $(y_{i}\mathbf{a}_{i}^{T}\mathbf{x}-1)^{2}=(y_{i}-\mathbf{a}_{i}^{T}\mathbf{x})^{2}.$

Thus, the estimator $\widehat{\mathbf{x}}$ is minimizing the sum of squares of the residuals:

[TABLE]

For the choice $\ell(t)=(t-1)^{2}$, it turns out that we can solve the equations in (9) in closed form. The final result is summarized in the corollary below.

Corollary IV.1** (Least-squares).**

Let Assumption 1 hold and $\delta>1$. Let $\widehat{\mathbf{x}}$ be as in (19). Then, in the limit of $m,n\rightarrow+\infty$, $m/n\rightarrow\delta$, Equations (12) and (13) hold with probability one with $\alpha$ and $\mu$ given as follows:

[TABLE]

Proof.

In order to get the values of $\alpha$ and $\mu$ as in the statement of the corollary, we show how to simplify Equations (9) for $\ell(t)=(t-1)^{2}$. In this case, the proximal operator admits a simple expression:

[TABLE]

Also, $\ell^{{}^{\prime}}(t)=2(t-1)$. Substituting these in (10a) gives the formula for $\mu$ as follows:

[TABLE]

where we have also used from (8) that $\mathbb{E}[S^{2}]=1$, $\mathbb{E}[YS]=(1-2\varepsilon)\sqrt{\frac{2}{\pi}}$ and $G$ is independent of $S$. Also, since $\ell^{{}^{\prime\prime}}(t)=2$, direct application of (11) gives

[TABLE]

Finally, substituting the value of $\lambda$ in (10b) we obtain the desired value for $\alpha$ as follows

[TABLE]

∎

Remark 7* (Least-squares: One-bit vs signed measurements).*

On the one hand, Corollary IV.1 shows that least-squares for (noisy) one-bit measurements lead to an estimator that satisfies

[TABLE]

where $\mu$ is as in (20) and $\tau^{2}:=1-(1-2\varepsilon)\frac{2}{\pi}$. On the other hand, it is well-known (e.g., see references in [30, Sec. 5.1]) that least-squares for (scaled) linear measurements with additive Gaussian noise (i.e. $y_{i}=\rho\mathbf{a}_{i}^{T}\mathbf{x}_{0}+\sigma z_{i}$, $z_{i}\sim\mathcal{N}(0,1)$) leads to an estimator that satisfies

[TABLE]

Direct comparison of (22) to (23) suggests that least-squares with one-bit measurements performs the same as if measurements were linear with scaling factor $\rho=\mu/\|\mathbf{x}_{0}\|_{2}$ and noise variance $\sigma^{2}=\tau^{2}=\alpha^{2}(\delta-1)$. This worth-mentioning conclusion is not new; it was proved in [6, 23, 29]. We elaborate on the relation to this prior work in the following remark.

Remark 8* (Prior work).*

There is a lot of recent work on the use of least-squares-type estimators for recovering signals from nonlinear measurements of the form $y_{i}=f(\mathbf{a}_{i}^{T}\mathbf{x}_{0})$ with Gaussian vectors $\mathbf{a}_{i}$. The original work that suggests least-squares as a reasonable estimator in this setting is due to Brillinger [6]. In his 1982 paper, Brillinger studied the problem in the classical statistics regime (aka $n$ is fixed not scaling with $m\rightarrow+\infty$) and he proved for the least-squares solution satisfies

[TABLE]

where

[TABLE]

and the expectations are with respect to $S$ and possible randomness of $f$. Evaluating (24) for $f(S)={\rm{BSC}}_{\varepsilon}(\mathrm{sign}(S))$ leads to the same values for $\mu$ and $\tau^{2}$ in (22). In other works, (22) for $\delta\rightarrow+\infty$ indeed recovers Brillinger’s result. The extension of Brillinger’s original work to the high-dimensional setting (both $m,n$ large) was first studied by Plan and Vershynin [23], who derived (non-sharp) non-asymptotic upper bounds on the performance of constrained least-squares (such as the Lasso). Shortly after, [29] extended this result to sharp asymtpotic predictions and to regularized least-squares. In particular, Corollary IV.1 is a special case of the main theorem in [29]. Several other interesting extensions of the result by Plan and Vershynin have recently appeared in the literature, e.g., [14, 16, 15, 32]. However, [29] is the only one to give results that are sharp in the flavor of this paper. Our work, extends the result of [29] to general loss functions beyond least-squares. The techniques of [29] that have guided the use of the CGMT in our context have also been recently applied in [9] in the context of phase-retrieval.

IV-B Least-absolute deviations

By choosing $\ell(t)=|t-1|$ in (4), we obtain a least-absolute deviations estimate. Again, since $y_{i}=\pm 1$, it holds for all $i$ that $|y_{i}\mathbf{a}_{i}^{T}\mathbf{x}-1|=|y_{i}-\mathbf{a}_{i}^{T}\mathbf{x}|.$ Thus, this choice of loss function leads to residuals:

[TABLE]

As in Section IV-A, for $\ell(t)=|t-1|$ the proximal operator admits a simple expression, as follows:

[TABLE]

where

[TABLE]

is the standard soft-thresholding function.

IV-C Hinge-loss

We obtain the hinge-loss estimator in by setting $\ell(t)=\max(1-t,0)$ in (4). Similar to Section IV-B, the proximal operator of the hinge-loss can be expressed in terms of the soft-thresholding function as follows:

[TABLE]

As already mentioned in Remark 3, the set of minimizers of the hinge-loss is bounded (required by Theorem III.1) only for $\delta>\delta^{\star}_{\varepsilon}$ where $\delta^{\star}_{\varepsilon}$ is the value of the threshold in (15). Our numerical simulations in Figures 4 and 5 suggest that hinge-loss is robust to measurement corruptions, as for moderate to large values of $\delta$ it outperforms the LS and the LAD estimators. Theorem III.1 opens the way to analytically confirm such conclusions, which is an interesting future direction.

IV-D Numerical simulations

We present numerical simulations that validate the predictions of Theorems III.1 and III.2 . For the numerical experiments, we generate random measurements according to (3) and Assumption 1. Without loss of generality (due to rotational invariance of the Gaussian measure) we set $\mathbf{x}_{0}=[1,0,...,0]^{T}$. We then obtain estimates $\widehat{\mathbf{x}}_{\ell}$ of $\mathbf{x}_{0}$ by numerically solving (4) and measuring performance by the correlation value ${\rm{corr}}\left(\,{\hat{\mathbf{x}}_{\ell}}\,;\,{\mathbf{x}_{0}}\,\right)$. Throughout the experiments, we set $n=128$ and the recorded values of correlation in Figures 3–5 are averages over $25$ independent experiments. The theoretical curves for the correlation are computed based on Theorem III.1. We solve the system of equations in (9) by the fixed-point iteration method described in Remark 5. The expectations involved in (9) are evaluated with Monte-Carlo estimation using $10^{5}$ independent samples. Numerical upper bounds in Figures 3-5 are derived according to Remark 6.

Comparisons between theoretical and simulated values for LAD and LS estimators along with the upper bound on correlation are illustrated in Figure 3 for the noiseless case. Note that for $\varepsilon=0$, the hinge-loss has an unbounded set of minimizers for all values of $\delta$ (thus, Theorem III.1 is not applicable). In Figure 4, the probability of error $\epsilon$ is increased to $0.1$. Note that in this setting hinge-loss estimator exists for $\delta>\delta^{*}_{0.1}\approx 3$ and that it outperforms LAD and LS estimators for large values of $\delta$.

In Figure 5 we present similar results for $\epsilon=0.25$. As it is evident from Figures 4 and 5, the best estimator is varying based on the value of $\delta$ and $\varepsilon$. This further emphasizes the impact of studying the accuracy of estimators while we do not restrict ourselves to a specific loss function.

V Conclusion

This paper derives sharp asymptotic performance guarantees for a wide class of convex optimization based estimators for recovering a signal from corrupted one-bit measurements. Our general result includes as a special case the least-squares estimator that was previously studied in [29]. Beyond that, it applies to other popular estimators such as the LAD, Hinge-loss, logistic loss, etc. One natural and interesting research direction is finding the optimal loss function $\ell(\cdot)$ in (4). In view of Theorem III.1, this boils down to finding $\ell(\cdot)$ that minimizes the ratio $\alpha/\mu$ of the parameters $\alpha$ and $\mu$ that solve the system of equations in (9). For this purpose, it might also be important to derive necessary and sufficient conditions that guarantee (9) has a unique solution.

Appendix A Analysis

In this section we provide a proof sketch of Theorem III.1. The main technical tool that facilitates our analysis is the convex Gaussian min-max theorem (CGMT), which is an extension of Gordon’s Gaussian min-max inequality (GMT). We introduce the necessary background on the CGMT in A-A.

A-A Technical tool: CGMT

A-A1 Gordon’s Min-Max Theorem (GMT)

The Gordon’s Gaussian comparison inequality [17] compares the min-max value of two doubly indexed Gaussian processes based on how their autocorrelation functions compare. The inequality is quite general (see [17]), but for our purposes we only need its application to the following two Gaussian processes:

[TABLE]

where: $\mathbf{G}\in\mathbb{R}^{m\times n}$, $\mathbf{g}\in\mathbb{R}^{m}$, $\mathbf{h}\in\mathbb{R}^{n}$, they all have entries iid Gaussian; the sets $\mathcal{S}_{\mathbf{w}}\subset\mathbb{R}^{n}$ and $\mathcal{S}_{\mathbf{u}}\subset\mathbb{R}^{m}$ are compact; and, $\psi:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$. For these two processes, define the following (random) min-max optimization programs, which (following [30]) we refer to as the primary optimization (PO) problem and the auxiliary optimization (AO) – for purposes that will soon become clear.

[TABLE]

According to Gordon’s comparison inequality, for any $c\in\mathbb{R}$, it holds:

[TABLE]

In other words, a high-probability lower bound on the AO is a high-probability lower bound on the PO. The premise is that it is often much simpler to lower bound the AO rather than the PO. To be precise, (29) is a slight reformulation of Gordon’s original result proved in [31] (see therein for details).

A-A2 Convex Gaussian Min-Max Theorem (CGMT)

The proof of Theorem III.1 builds on the CGMT [31]. For ease of reference we summarize here the essential ideas of the framework following the presentation in [30]; please see [30, Section 6] for the formal statement of the theorem and further details. The CGMT is an extension of the GMT and it asserts that the AO in (28b) can be used to tightly infer properties of the original (PO) in (28a), including the optimal cost and the optimal solution. According to the CGMT [30, Theorem 6.1], if the sets $\mathcal{S}_{\mathbf{w}}$ and $\mathcal{S}_{\mathbf{u}}$ are convex and $\psi$ is continuous convex-concave on $\mathcal{S}_{\mathbf{w}}\times\mathcal{S}_{\mathbf{u}}$, then, for any $\nu\in\mathbb{R}$ and $t>0$, it holds

[TABLE]

In words, concentration of the optimal cost of the AO problem around $\mu$ implies concentration of the optimal cost of the corresponding PO problem around the same value $\mu$. Moreover, starting from (30) and under strict convexity conditions, the CGMT shows that concentration of the optimal solution of the AO problem implies concentration of the optimal solution of the PO to the same value. For example, if minimizers of (28b) satisfy $\lVert\mathbf{w}^{\ast}(\mathbf{g},\mathbf{h})\rVert_{2}\to\zeta^{\ast}$ for some $\zeta^{\ast}>0$, then, the same holds true for the minimizers of (28a): $\lVert\mathbf{w}^{\ast}(\mathbf{G})\rVert_{2}\to\zeta^{\ast}$ [30, Theorem 6.1(iii)]. Thus, one can analyze the AO to infer corresponding properties of the PO, the premise being of course that the former is simpler to handle than the latter.

A-B Applying the CGMT to (7)

In this section, we show how to apply the CGMT to (7). For convenience, we drop the subscript $\ell$ from $\widehat{\mathbf{x}}_{\ell}$ and simply write

[TABLE]

where the measurements $y_{i},~{}i\in[m]$ follow (3). By rotational invariance of the Gaussian distribution of the measurement vectors $\mathbf{a}_{i},~{}i\in[m]$, we assume without loss of generality that $\mathbf{x}_{0}=[1,0,...,0]^{T}$. Denoting $y_{i}\mathbf{a}_{i}^{T}\mathbf{x}$ by $u_{i}$, (31) is equivalent to the following min-max optimization:

[TABLE]

Now, let us define

[TABLE]

such that $s_{i}$ and $x_{1}$ are the first entries of $\mathbf{a}_{i}$ and $\mathbf{x}$, respectively. Note that in this new notation (6) becomes:

[TABLE]

and

[TABLE]

where we denote $\widehat{\mathbf{x}}=[\widehat{\mathbf{x}}_{1};{\widetilde{\widehat{\mathbf{x}}}}]$. Also, (32) is written as

[TABLE]

or, in matrix form:

[TABLE]

where $\mathbf{D}_{\mathbf{y}}:={\rm{diag}}(y_{1},y_{2},...,y_{m})$ is a diagonal matrix with $y_{1},y_{2},...y_{m}$ on the diagonal, $\mathbf{s}=[s_{1},\ldots,s_{m}]^{T}$ and $\tilde{\mathbf{A}}$ is an $m\times(n-1)$ matrix with rows $\tilde{\mathbf{a}}_{i}^{T},~{}i\in[m]$.

In (35) we recognize that the first term has the bilinear form required by the GMT in (28a). The rest of the terms form the function $\psi$ in (28a): they are independent of $\tilde{\mathbf{A}}$ and convex-concave as desired by the CGMT. Therefore, we have expressed (31) in the desired form of a PO and for the rest of the proof we will analyze the probabilistically equivalent AO problem. In view of (28b), this is given as follows,

[TABLE]

where as in (28b) $\mathbf{g}\sim\mathcal{N}(0,I_{m})$ and $\mathbf{h}\sim\mathcal{N}(0,I_{n-1})$.

A-C Analysis of the Auxiliary Optimization

Here, we show how to analyze the AO in (36) 222There are several technical details in the proof that we omit from this proof sketch. This includes: boundedness of the constraint sets in (36); changing the order of min-max when optimizing over the direction of $\tilde{\mathbf{x}}$ in (37); and, uniform convergence in going from (39) to (40).. The steps are similar and follow the recipe prescribed in [31, 30]. To begin with, note that $y_{i}\in{\pm 1}$, therefore $\mathbf{D}_{\mathbf{y}}\mathbf{g}\sim\mathcal{N}(0,I_{m})$ and $\left\lVert\mathbf{D}_{\mathbf{y}}\boldsymbol{\beta}\right\rVert_{2}=\left\lVert\boldsymbol{\beta}\right\rVert_{2}$. Let us denote

[TABLE]

We can simplify (36) by optimizing over the direction of $\tilde{\mathbf{x}}$. This leads to the following:

[TABLE]

Next, let $\gamma:=\frac{\left\lVert\boldsymbol{\beta}\right\rVert_{2}}{\sqrt{m}}$ and optimize over the direction of $\boldsymbol{\beta}$ to yield

[TABLE]

To continue, we utilize the fact that for all $x\in\mathbb{R}$, $\min_{\tau>0}\frac{\tau}{2}+\frac{x^{2}}{2\tau m}=\frac{x}{\sqrt{m}}$. With this trick, the optimization over $\mathbf{u}$ becomes separable over its coordinates and (38) can be rewritten as

[TABLE]

Equivalently, we express this in the following convenient form:

[TABLE]

where recall the definition of the Moreau envelope:

[TABLE]

As to now, we have reduced the AO into a random min-max optimization over only four scalar variables in (39). For fixed $\mu,\alpha,\tau,\gamma$, direct application of the weak law of large numbers, shows that the objective function of (39) converges in probability to the following as $m,n\rightarrow\infty$ and $\frac{m}{n}=\delta$:

[TABLE]

where $G,S\sim\mathcal{N}(0,1)$ and $Y\sim 2\mathrm{Bernoulli}(f(S))-1$ (in view of (33)). Based on that, it can be shown (see [30, 9] for similar arguments; details are deferred to the long version of the paper) that the random optimizers $\alpha_{n}$ and $\mu_{n}$ of (39) converge to the deterministic optimizers $\alpha$ and $\mu$ of the following (deterministic) optimization problem (whenever these are bounded as the statement of the theorem requires):

[TABLE]

At this point, recall that $\alpha$ represents the norm of $\tilde{\mathbf{x}}$ and $\mu$ the value of $x_{1}$. Thus, in view of (i) (34), (ii) the equivalence between the PO and the AO, and, (iii) our derivations thus far we have that

[TABLE]

where $\mu$ and $\alpha$ are the minimizers in (40). The three equations in (9) are derived by the first-order optimality conditions of the optimization in (40). We show this next.

A-D First-order optimality conditions

By direct differentiation, the first order optimality conditions of of the min-max optimization in (40) are as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Next, we show how these equations simplify to the following system of equation:

[TABLE]

Denote $\lambda:=\frac{\tau}{\gamma}$. First, (45a) is followed by equation (41). Second, substituting $\gamma$ from (43) in (44) yields $\tau=\frac{\alpha}{\sqrt{\delta}}$ or $\gamma=\frac{\alpha}{\lambda\sqrt{\delta}}$, which together with (42) leads to (45c). Finally, (45b) can be obtained by substituting $\gamma=\frac{\alpha}{\lambda\sqrt{\delta}}$ in (43) and using the relation:

[TABLE]

As already mentioned in Section II, we focus on the non-regularized loss functions in this paper. Setting $r=0$ in (45) will give the desired system of equations in (9)