Deep Signal Recovery with One-Bit Quantization

TL;DR

This paper introduces DeepRec, a deep learning model that unfolds an inference algorithm for efficient and accurate reconstruction of high-dimensional signals from one-bit noisy measurements, advancing signal recovery techniques.

Contribution

It proposes a novel deep unfolding approach for one-bit signal recovery, combining model-based methods with deep learning for improved performance.

Findings

Enhanced accuracy over traditional methods

Reduced computational complexity

Effective high-dimensional signal reconstruction

Abstract

Machine learning, and more specifically deep learning, have shown remarkable performance in sensing, communications, and inference. In this paper, we consider the application of the deep unfolding technique in the problem of signal reconstruction from its one-bit noisy measurements. Namely, we propose a model-based machine learning method and unfold the iterations of an inference optimization algorithm into the layers of a deep neural network for one-bit signal recovery. The resulting network, which we refer to as DeepRec, can efficiently handle the recovery of high-dimensional signals from acquired one-bit noisy measurements. The proposed method results in an improvement in accuracy and computational efficiency with respect to the original framework as shown through numerical analysis.

Equations

$$\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{n},$$

$$\mathbf{r}\triangleq \text{sign}(\mathbf{y}-\mathbf{\tau }),$$

$$p_i=\text{Prob}\{{\mathbf{h}}_i^T\mathbf{x}+n_i-{\tau }_i>0\}=Q\left(\frac{{\tau }_i-{\mathbf{h}}_i^T\mathbf{x}}{{\sigma }_i}\right),$$

$$p(r_i)=Q\left(\frac{r_i}{{\sigma }_i}\left({\tau }_i-{\mathbf{h}}_i^T\mathbf{x}\right)\right),$$

$$\mathcal{L}(\mathbf{x})=p(\mathbf{r}|\mathbf{x})$$

$$=\sum _{i=1}^N\text{log}\left\{Q\left(\frac{r_i}{{\sigma }_i}\left({\tau }_i-{\mathbf{h}}_i^T\mathbf{x}\right)\right)\right\},$$

$$\hat{\mathbf{x}}=\underset{\mathbf{x}}{\mathrm{argmax}}\mathcal{L}(\mathbf{x}).$$

$${\nabla }_{\mathbf{x}}\mathcal{L}(\mathbf{x})=0,$$

$${\nabla }_{\mathbf{x}}\mathcal{L}(\mathbf{x})=$$

$$\mathbf{\eta }(\mathbf{x})\triangleq \frac{Q^{\prime }(\mathbf{x})}{Q(\mathbf{x})},$$

$${\nabla }_{\mathbf{x}}\mathcal{L}(\mathbf{x})=-{\mathbf{H}}^T\tilde{\mathbf{\Omega }}\mathbf{\eta }\left(\tilde{\mathbf{\Omega }}(\mathbf{\tau }-\mathbf{H}\mathbf{x})\right).$$

$${\nabla }_{\mathbf{x}}\mathcal{L}(\mathbf{x})=-{\mathbf{H}}^T\tilde{\mathbf{\Omega }}\mathbf{\eta }\left(\tilde{\mathbf{\Omega }}(\mathbf{\tau }-\mathbf{H}\mathbf{x})\right)=0.$$

$${\mathbf{x}}^{(k+1)}$$

$$={\mathbf{x}}^{(k)}-{\delta }^{(k)}{\mathbf{H}}^T\tilde{\mathbf{\Omega }}\mathbf{\eta }\left(\tilde{\mathbf{\Omega }}(\mathbf{\tau }-\mathbf{H}{\mathbf{x}}^{(k)})\right),$$

$${\mathbf{z}}^{(k)}={\mathbf{W}}_{1k}\tilde{\mathbf{\Omega }}\mathbf{\tau }-{\mathbf{W}}_{2k}\mathbf{H}{\mathbf{x}}^{(k)}+{\mathbf{b}}_{1k},$$

$${\mathbf{p}}^{(k)}=\mathbf{\eta }\left({\mathbf{z}}^{(k)}\right),$$

$${\mathbf{t}}^{(k)}={\mathbf{H}}^T\tilde{\mathbf{\Omega }}{\mathbf{p}}^{(k)},$$

$${\mathbf{x}}^{(k+1)}=f\left({\mathbf{W}}_{3k}\left[\begin{array}{c}{\mathbf{x}}^{(k)}\\ {\mathbf{t}}^{(k)}\end{array}\right]+{\mathbf{b}}_{2k}\right),$$

$$\mathbf{\Xi }=\{{\mathbf{W}}_{1k},{\mathbf{W}}_{2k},{\mathbf{W}}_{3k},{\mathbf{b}}_{1k},{\mathbf{b}}_{2k}{\}}_{k=1}^K.$$

$$\underset{\mathbf{\Xi }}{\min }\mathcal{R}(\mathbf{x};{\mathbf{\Psi }}_{\mathbf{\Xi }}(\mathbf{r},\mathbf{H},\mathbf{\tau })).$$

$$\mathcal{R}\left(\mathbf{x};{\mathbf{\Psi }}_{\mathbf{\Xi }}(\mathbf{r},\mathbf{H},\mathbf{\tau })\right)=||\mathbf{x}-{\mathbf{\Psi }}_{\mathbf{\Xi }}(\mathbf{r},\mathbf{H},\mathbf{\tau })|{|}_2^2,$$

Full text

**Index Terms— ** Deep learning, deep unfolding, MIMO communications, big data, machine learning, neural network, maximum likelihood, one-bit quantization

1 Introduction

Quantization of signals of interest is an integral part of all modern digital signal processing applications such as sensing, communication, and inference. In an ideal hardware implementation of a quantization system, a high-resolution analog-to-digital converter (ADC) with $b$-bit resolution and sampling frequency of $f_{s}$ samples the original analog signal and maps the obtained samples into a discrete state space of size $2^{b}f_{s}$. Generally, a large number of bits is required to obtain an accurate digital representation of the analog signal. In such a case, the quantization process has negligible impact on the performance of algorithms which were typically developed on the assumptions of infinite precision samples, and thus, the high-resolution (in terms of amplitude) quantization process can be directly modeled as an additive noise source. However, a crucial obstacle with modern ADCs is that their power consumption, manufacturing cost, and chip area grows exponentially with their resolution $b$ [1, 2, 3].

The required high sampling data rate of ADCs used in next generataion communications systems is another obstacle that must be tackled in such applications. For instance, the promising millimeter wave (mmWave) multiple-input multiple output (MIMO) communication technology requires a very large bandwidth, and the corresponding sampling rate of the ADCs must increase accordingly. However, manufacturing ADCs with high-resolution (e.g., more than 8 bits) and high sampling rate are extremely costly and may not be available. Moreover, in other applications such as spectral sensing and cognitive radio, which require extremely high sampling rates, the cumulative cost and power consumption of using high-resolution and extremely fast ADCs are typically prohibitive and impractical. Hence, when signals across a wide frequency band are of interest, a fundamental trade-off between sampling rate, amplitude quantization precision, cost, and power consumption is encountered. An immediate solution to such challenges is to use low-resolution, and specifically one-bit, ADCs. The use of one-bit signed measurements, and more specifically one-bit ADCs, allows for an extremely high sampling rate at a low cost and low power consumption. From a sampling viewpoint, the most extreme case of quantization is to use only one bit per sample. More precisely, one-bit sampling can be seen as a process through which we repeatedly compare the amplitude of a signal (at each sample) to some reference threshold level and use only one bit to convey whether the signal amplitude resides above or below that threshold. Due to its appealing sampling properties, the problem of recovering a signal from its one-bit measurements has attracted a great deal of interest over the past few years [4, 5, 6, 7, 8]. Therefore, it is vital to develop algorithms that can deal with low-resolution samples for different applications.

The fields of machine learning (ML), and more particularly deep learning, are impacting various fields of engineering and have recently attracted a great deal of attention in tackling long-standing signal processing problems. The advent of low-cost specialized powerful computing resources (e.g., GPUs, and more recently TPUs) and the continually increasing amount of massive data generated by the human population and machines, in conjunction with the new optimization and learning methods, have paved the way for deep neural networks (DNNs) and machine learning-based models to prove their effectiveness in many engineering areas (see, e.g., [9, 10, 11] and the references therein).

The main advantage of the deep learning-based model herein is that it employs several non-linear transformations to obtain an abstract representation of the underlying data. Model-based machine learning frameworks (e.g., probabilistic graphical models) incorporate prior knowledge of the system parameters into the inference process. A recent promising approach in bridging the gap between deep learning-based and model-based methods is the paradigm of deep unfolding [12]. Particularly, iterations of a conventional recursive algorithm, such as fast iterative soft thresholding algorithm (FISTA), projected gradient descent, and approximate message passing (AMP), can be used as a baseline to design the architecture of a deep network with trainable parameters specifically customized to the problem of interest. Such a methodology results in an improvement in accuracy, and computational efficiency of the original framework. The deep unfolding method has already shown remarkable performance improvement in a wide range of applications such as MIMO communications [13, 14], multi-channel source separation [15], and sparse inverse problems [16, 17].

In this paper, we consider the general problem of high-dimensional signal recovery from random one-bit measurements. Specifically, we propose an efficient signal recovery framework based on the deep unfolding technique that has the advantage of low-complexity and near-optimal performance compared to traditional methods. Our proposed inference framework has a wide range of applications in the areas of wireless communications, detection and estimation, and sensing.

2 Problem Formulation

We begin by considering a general linear signal acquisition and one-bit quantization model with time-varying thresholds, described as follows:

[TABLE]

where ${\boldsymbol{\tau}}=[\tau_{1},\dots,\tau_{M}]^{T}$ denotes the vector of one-bit quantization thresholds, ${\boldsymbol{y}}\in\mathds{R}^{M}$ denotes the received signal prior to quantization, ${\boldsymbol{H}}\in\mathds{R}^{M\times N}$ denotes the sensing matrix, ${\boldsymbol{x}}\in\mathds{R}^{N}$ denotes the multidimensional unknown vector to be recovered, and ${\boldsymbol{n}}\sim\mathcal{N}(\mathbf{0},{\boldsymbol{C}})$ denotes the zero-mean Gaussian noise with a known covariance matrix ${\boldsymbol{C}}=\textbf{Diag}(\sigma_{1}^{2},\dots,\sigma_{M}^{2})$. Furthermore, $\text{sign}(\cdot)$ denotes the signum function applied element-wise on the vector argument.

The above model covers a wide range of applications. For instance, the described model (1)-(2) can be used in MIMO communication systems in which ${\boldsymbol{H}}$ is the channel matrix, ${\boldsymbol{x}}$ is the signal sent by the transmitter, ${\boldsymbol{n}}$ is the additive Gaussian noise in the system, and the base station is equipped with one-bit ADCs, where the goal is to recover the transmitted symbols from ${\boldsymbol{r}}$.

2.1 Maximum Likelihood Estimator Derivation

Given the knowledge of the sensing matrix ${\boldsymbol{H}}$, noise covariance ${\boldsymbol{C}}$, and the corresponding quantization thresholds ${\boldsymbol{\tau}}$, our goal is to recover the original (likely high-dimensional) signal ${\boldsymbol{x}}$ from the one-bit random measurements ${\boldsymbol{r}}$. In such a scenario, each binary observation $\{r_{i}\}_{i=1}^{N}$ follows a Bernoulli distribution with parameter $p_{i}$, given by:

[TABLE]

where $Q(x)=1-\Phi(x)$ with $\Phi(x)$ representing the cumulative distribution function (cdf) of a standard Gaussian distribution and ${\boldsymbol{h}}_{i}^{T}$ denotes the $i$-th row of the matrix ${\boldsymbol{H}}$. In particular, the probability mass function (pmf) of each binary observation can be compactly expressed as:

[TABLE]

where $r_{i}\in\{-1,+1\}$. Therefore, the log-likelihood of the quantized observations ${\boldsymbol{r}}$ given the unknown vector ${\boldsymbol{x}}$ can be expressed as:

[TABLE]

where $\text{log}\left\{\cdot\right\}$ denotes the natural logarithm. As a result, the maximum likelihood (ML) estimation of ${\boldsymbol{x}}$ can be obtained as

[TABLE]

Observe that the maximum likelihood estimator $\hat{{\boldsymbol{x}}}$ has to satisfy the following condition:

[TABLE]

where the gradient of the log-likelihood function with respect to the unknown vector ${\boldsymbol{x}}$ can be derived as follows:

[TABLE]

where $Q^{\prime}(x)=-\frac{1}{\sqrt{2\pi}}\text{exp}\left(-x^{2}/2\right)$. It can be observed from (9) that the gradient of the log-likelihood function is a linear combination of the rows of the sensing matrix ${\boldsymbol{H}}$. Let $\boldsymbol{\eta}\colon\mathds{R}^{M}\mapsto\mathds{R}^{M}$ be a non-linear function defined as follows:

[TABLE]

where the functions $Q(\cdot)$, $Q^{\prime}(\cdot)$, and the division, are applied element-wise on the vector argument ${\boldsymbol{x}}$. In addition, let $\boldsymbol{\Omega}=\textbf{Diag}(r_{1},\dots,r_{M})$ be a diagonal matrix containing the one-bit observations and $\tilde{\boldsymbol{\Omega}}=\boldsymbol{\Omega}{\boldsymbol{C}}^{-\frac{1}{2}}$ be the semi-whitened version of the one-bit matrix $\boldsymbol{\Omega}$. Then, the gradient of the likelihood function in (9) can be compactly written as follows:

[TABLE]

Recall that the ML estimator $\hat{{\boldsymbol{x}}}$ must satisfy the condition given in (8), i.e.,

[TABLE]

Other than certain low-dimensional cases, finding a closed-form expression for $\hat{{\boldsymbol{x}}}$ that satisfies (12) is a difficult task [18, 19, 20]. Therefore, we resort to iterative methods in order to find the ML estimate, i.e., to solve (7).

In this paper, the well-known gradient ascent method is employed to iteratively solve (7). Namely, given an initial point ${\boldsymbol{x}}^{(0)}$, the update equation at each iteration is given by:

[TABLE]

where $\delta^{(k)}$ is the step size at the $k$-th iteration. The obtained maximum likelihood estimator derived from the signal model, and the corresponding optimization steps, can be unfolded into a multi-layer deep neural network, which improves the accuracy and computational effciency of the original framework.

In the next section, we unfold the above iterations into the layers of a deep neural network where each layer denotes one iteration of the above optimization method. Interestingly, we fix the complexity budget of the inference framework (via fixing the number of layers), and apply the gradient descent method to yield the most accurate estimation of the parameter in at most $K$ iterations.

3 Signal Recovery via Deep Unfolding

Conventionally, first-order optimization methods, such as gradient descent algorithms, have slow convergence rate, and thus take a large number of iterations to converge to a solution. Herein, we are interested in finding a good solution under the condition that the complexity of the inference algorithm is fixed. This is important since, via unfolding the optimization algorithm, we fix the computational complexity of the inference model (a DNN with $K$ layers in such a case) and optimize the parameters of the network to find the best possible estimator with a fixed-complexity constraint. Below, we introduce DeepRec, our deep learning based signal recovery framework which is designed based on the iterations of the form (14), to find the maximum likelihood estimation of the unknown parameter.

*—*The DeepRec Architecture. The construction of DeepRec involves the unfolding of $k=1,\dots,K,$ iterations each of which are of the form (14), as the layers of a deep neural network. Particularly, each step of the gradient descent method depends on the previous signal estimate ${\boldsymbol{x}}^{(k)}$, the step size $\delta^{(k)}$, the scaled one-bit matrix $\tilde{\boldsymbol{\Omega}}$, the sensing matrix ${\boldsymbol{H}}$, and the threshold vector ${\boldsymbol{\tau}}$. In addition, the form of the gradient vector (11) makes it convenient and insightful to unfold the iterations onto the layers of a DNN in that each iteration of the gradient descent method is a linear combination of the system paramteres followed by a non-linearity. The $k$-th layer of DeepRec can be characterized via the following operations and variables:

[TABLE]

where ${\boldsymbol{x}}^{(1)}=\mathbf{0}$, $f(\cdot)$ denotes a non-linear activation function where in this work we consider $f(x)\triangleq\text{ReLU}(x)=\text{max}\{0,x\}$, and the goal is to optimize the DNN parameters, described as follows:

[TABLE]

The proposed DeepRec architecture with $L$ layers can be interpreted as a class of estimator functions $\boldsymbol{\Psi}_{\boldsymbol{\Xi}}({\boldsymbol{r}},{\boldsymbol{H}},{\boldsymbol{\tau}})$ parametrized by $\boldsymbol{\Xi}$ to estimate the unknown parameter ${\boldsymbol{x}}$ given the system parameters. In order to find the best estimator function $\boldsymbol{\Psi}_{\boldsymbol{\Xi}}({\boldsymbol{r}},{\boldsymbol{H}},{\boldsymbol{\tau}})$ associated with our problem, we conduct a learning process via minimizing a loss function $\mathcal{R}({\boldsymbol{x}};\boldsymbol{\Psi}_{\boldsymbol{\Xi}}({\boldsymbol{r}},{\boldsymbol{H}},{\boldsymbol{\tau}}))$, i.e.,

[TABLE]

In this work, we employ the following least squares (LS) loss function:

[TABLE]

where during the training phase, we synthetically generate the system parameters $\boldsymbol{\Theta}=\{{\boldsymbol{x}},{\boldsymbol{r}},{\boldsymbol{H}},{\boldsymbol{\tau}}\}$ according to their statistical model.

4 Numerical Results

We now demonstrate the performance of the proposed DeepRec framework for the problem of one-bit signal recovery. The proposed framework was implemented using the TensorFlow library [21], with the ADAM stochastic optimizer [22] and an exponential decaying step size. In the learning process of the network, we employed the batch training method with a batch size of $500$ at each epoch and we performed the training for $2000$ epochs. In all of the simulations, we assumed $N=3$, i.e., ${\boldsymbol{x}}\in\mathds{R}^{3}$, and we used the normalized mean square error (NMSE) defined as $NMSE=||{\boldsymbol{x}}-\hat{\boldsymbol{x}}||_{2}^{2}/||{\boldsymbol{x}}||_{2}^{2}$, for the performance metric.

The training was performed based on the data generated via the following model. Each element of the vector ${\boldsymbol{x}}$ is assumed to be i.i.d and uniformly distributed, i.e., ${\boldsymbol{x}}\sim\mathcal{U}(\delta_{l}^{\boldsymbol{x}},\delta_{u}^{{\boldsymbol{x}}})$. The sensing matrix is assumed to be fixed and follow a Normal distribution, where we consider ${\boldsymbol{H}}\sim~{}\mathcal{N}(\mathbf{0},{\boldsymbol{I}})$. The quantization thresholds were also generated from a uniform distribution, ${\boldsymbol{\tau}}\sim\mathcal{U}(\delta_{l}^{\boldsymbol{\tau}},\delta_{u}^{{\boldsymbol{\tau}}})$, where the lower and upper bound of the distribution is chosen in a fashion that at least covers the domain of ${\boldsymbol{x}}$. The noise is assumed to be independent from one sample to another and follows a Normal distribution, where the variance of each corresponding noise element is different, e.g., the noise covariance ${\boldsymbol{C}}=~{}\textbf{Diag}(\sigma_{1}^{2},\dots,\sigma_{M}^{2})$, with $\sigma_{i}^{2}\sim\mathcal{U}(\delta_{1}^{{\boldsymbol{n}}},\delta_{M}^{{\boldsymbol{n}}})$. Note that we trained the network over a wide range of noise powers in order to make the DeepRec network more robust to noise.

Fig. 1(a) demonstrates the performance of the DeepRec network for different numbers of layers $K$. It can be observed that the averaged NMSE decreases dramatically as the number of layers increases. Such a result is also expected as each layer corresponds to one iteration of originial optimization algorithm. Thus, as the number of layers increases, the output of the network will converge to a better estimation as well.

Fig. 1(b) demonstrates the performance of the proposed DeepRec architecture and the original Gradient Descent method of (14) in terms of averaged NMSE for different numbers of one-bit samples $M$. In this simulation, we implemented the DeepRec network with $K=90$ layers. It can be clearly seen from Fig. 1(b) that the proposed deep recovery architecture (DeepRec) significantly outperforms the original optimization method in terms of accuracy and provides a considerably better estimation than that of the gradient descent method for the same number of iterations/layers. As a fair comparison, we also assumed a fixed-step size of $\delta=0.01$ for the gradient descent method.

Fig. 1(c) shows a comparison of the computational cost (machine runtime) between the gradient descent method and the proposed DeepRec network for different numbers of one-bit samples $M$. It can be seen that our proposed method (DeepRec) has a significantly lower computational cost than that of the original optimization algorithm for our problem. Hence, making the DeepRec a good candidate for real-time signal processing or big data applications (the results were obtained on a standard PC with a quad-core 2.30GHz CPU and 4 GB memory).

5 Conclusion

We have considered the application of model-based machine learning, and specifically the deep unfolding technique, in the problem of recovering a high-dimensional signal from its one-bit quantized noisy measurements via random thresholding. We proposed a novel deep architecture, which we refer to as DeepRec, that was able to accurately perform the task of one-bit signal recovery. Our numerical results show that the proposed DeepRec network significantly improves the performance of traditional optimization methods both in terms of accuracy and efficiency.