Unveiling Bias Compensation in Turbo-Based Algorithms for (Discrete) Compressed Sensing

TL;DR

This paper explores turbo-based algorithms for discrete compressed sensing, revealing how bias compensation improves recovery of finite-valued sparse signals and providing a deeper understanding from a communications perspective.

Contribution

It introduces an analysis of turbo algorithms for discrete compressed sensing, highlighting bias unbiasing and proposing an improved algorithm.

Findings

Bias calculation equals unbiasing of biased estimates

Improved turbo algorithm enhances discrete signal recovery

Deeper understanding of algorithm behavior from communications perspective

Abstract

In Compressed Sensing, a real-valued sparse vector has to be recovered from an underdetermined system of linear equations. In many applications, however, the elements of the sparse vector are drawn from a finite set. Adapted algorithms incorporating this additional knowledge are required for the discrete-valued setup. In this paper, turbo-based algorithms for both cases are elucidated and analyzed from a communications engineering perspective, leading to a deeper understanding of the algorithm. In particular, we gain the intriguing insight that the calculation of extrinsic values is equal to the unbiasing of a biased estimate and present an improved algorithm.

Tables

Table 1. TABLE I: Comparison of the steps.

	First step	Second step
Processing	vectorwise (joint)	symbolwise (individual)
Assumption	$x$ Gaussian distributed	sparsity $s$, $x\in {𝒞}_0$
Estimation	linear/affine	non-linear
Estimated variable	signal $𝒙$	error $𝒆$

Equations

$$\displaystyle{\mathchoice{\mbox{\boldmath$\displaystyle y$}}{\mbox{\boldmath$\textstyle y$}}{\mbox{\boldmath$\scriptstyle y$}}{\mbox{\boldmath$\scriptscriptstyle y$}}}={\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}+{\mathchoice{\mbox{\boldmath$\displaystyle n$}}{\mbox{\boldmath$\textstyle n$}}{\mbox{\boldmath$\scriptstyle n$}}{\mbox{\boldmath$\scriptscriptstyle n$}}}\ ,$$

$$\displaystyle\hat{{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}}=\mathop{\mathrm{argmin}}_{\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}}\in\mathcal{C}_{0}^{L}}\|{\mathchoice{\mbox{\boldmath$\displaystyle y$}}{\mbox{\boldmath$\textstyle y$}}{\mbox{\boldmath$\scriptstyle y$}}{\mbox{\boldmath$\scriptscriptstyle y$}}}-{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}}\|_{2}^{2}\quad\text{s.t.}\quad\|\tilde{{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}}\|_{0}=s\ .$$

$$\displaystyle{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{post}}={\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}}+\frac{\bar{c}^{2}\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}}{\bar{c}^{2}\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}+\sigma_{n}^{2}}\cdot{\mathchoice{\mbox{\boldmath$\displaystyle C$}}{\mbox{\boldmath$\textstyle C$}}{\mbox{\boldmath$\scriptstyle C$}}{\mbox{\boldmath$\scriptscriptstyle C$}}}^{-1}{\mathchoice{\mbox{\boldmath$\displaystyle U$}}{\mbox{\boldmath$\textstyle U$}}{\mbox{\boldmath$\scriptstyle U$}}{\mbox{\boldmath$\scriptscriptstyle U$}}}^{\mathsf{T}}({\mathchoice{\mbox{\boldmath$\displaystyle y$}}{\mbox{\boldmath$\textstyle y$}}{\mbox{\boldmath$\scriptstyle y$}}{\mbox{\boldmath$\scriptscriptstyle y$}}}-{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}})\ ,$$

$$\displaystyle{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{post}}\approx{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}}+\frac{\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}}{\bar{c}^{2}\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}+\sigma_{n}^{2}}\cdot{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}^{\mathsf{T}}({\mathchoice{\mbox{\boldmath$\displaystyle y$}}{\mbox{\boldmath$\textstyle y$}}{\mbox{\boldmath$\scriptstyle y$}}{\mbox{\boldmath$\scriptscriptstyle y$}}}-{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}})\ .$$

$$\displaystyle{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{post}}={\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}}+{\mathchoice{\mbox{\boldmath$\displaystyle\Phi$}}{\mbox{\boldmath$\textstyle\Phi$}}{\mbox{\boldmath$\scriptstyle\Phi$}}{\mbox{\boldmath$\scriptscriptstyle\Phi$}}}_{dd}{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}^{\mathsf{T}}\left({\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}{\mathchoice{\mbox{\boldmath$\displaystyle\Phi$}}{\mbox{\boldmath$\textstyle\Phi$}}{\mbox{\boldmath$\scriptstyle\Phi$}}{\mbox{\boldmath$\scriptscriptstyle\Phi$}}}_{dd}{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}^{\mathsf{T}}+\sigma_{n}^{2}{\mathchoice{\mbox{\boldmath$\displaystyle I$}}{\mbox{\boldmath$\textstyle I$}}{\mbox{\boldmath$\scriptstyle I$}}{\mbox{\boldmath$\scriptscriptstyle I$}}}\right)^{-1}({\mathchoice{\mbox{\boldmath$\displaystyle y$}}{\mbox{\boldmath$\textstyle y$}}{\mbox{\boldmath$\scriptstyle y$}}{\mbox{\boldmath$\scriptscriptstyle y$}}}-{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}})\ ,$$

$$\displaystyle{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{post}}={\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}}+{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}^{\mathsf{T}}\left({\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}^{\mathsf{T}}+\frac{\sigma_{n}^{2}}{\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}}{\mathchoice{\mbox{\boldmath$\displaystyle I$}}{\mbox{\boldmath$\textstyle I$}}{\mbox{\boldmath$\scriptstyle I$}}{\mbox{\boldmath$\scriptscriptstyle I$}}}\right)^{-1}({\mathchoice{\mbox{\boldmath$\displaystyle y$}}{\mbox{\boldmath$\textstyle y$}}{\mbox{\boldmath$\scriptstyle y$}}{\mbox{\boldmath$\scriptscriptstyle y$}}}-{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}}).$$

$$\displaystyle{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{S}}^{\mathrm{pri}}={\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{ext}}=\sigma_{\mathrm{M},\mathrm{ext}}^{2}\,\left(\frac{{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{post}}}{\sigma_{\mathrm{M},\mathrm{post}}^{2}}-\frac{{\mathchoice{\mbox{\boldmath$\displaystyle x$}}{\mbox{\boldmath$\textstyle x$}}{\mbox{\boldmath$\scriptstyle x$}}{\mbox{\boldmath$\scriptscriptstyle x$}}}_{\mathrm{M}}^{\mathrm{pri}}}{\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}}\right)\ ,$$

$${\sigma }_{\mathrm{S},\mathrm{pri}}^2={\sigma }_{\mathrm{M},\mathrm{ext}}^2={\left(\frac{1}{{\sigma }_{\mathrm{M},\mathrm{post}}^2}-\frac{1}{{\sigma }_{\mathrm{M},\mathrm{pri}}^2}\right)}^{-1}.$$

$$\displaystyle{\mathchoice{\mbox{\boldmath$\displaystyle K$}}{\mbox{\boldmath$\textstyle K$}}{\mbox{\boldmath$\scriptstyle K$}}{\mbox{\boldmath$\scriptscriptstyle K$}}}=[K_{i,j}]={\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}^{\mathsf{T}}\left({\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}^{\mathsf{T}}+\frac{\sigma_{n}^{2}}{\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}}{\mathchoice{\mbox{\boldmath$\displaystyle I$}}{\mbox{\boldmath$\textstyle I$}}{\mbox{\boldmath$\scriptstyle I$}}{\mbox{\boldmath$\scriptscriptstyle I$}}}\right)^{-1}{\mathchoice{\mbox{\boldmath$\displaystyle A$}}{\mbox{\boldmath$\textstyle A$}}{\mbox{\boldmath$\scriptstyle A$}}{\mbox{\boldmath$\scriptscriptstyle A$}}}\ ,$$

$$x_{\mathrm{U},i}$$

$$=\frac{1}{K_{i,i}}\cdot x_{\mathrm{M},i}^{\mathrm{post}}-\left(\frac{1}{K_{i,i}}-1\right)\cdot x_{\mathrm{M},i}^{\mathrm{pri}}.$$

$${\sigma }_{\mathrm{M},\mathrm{post},i}^2$$

$$={\sigma }_{\mathrm{M},\mathrm{pri}}^2\cdot (1-K_{i,i}),$$

$${\sigma }_{\mathrm{U},i}^2={\sigma }_{\mathrm{M},\mathrm{pri}}^2\cdot \left(\frac{1}{K_{i,i}}-1\right).$$

$$x_{\mathrm{U},i}$$

$$=x_{\mathrm{M},i}^{\mathrm{ext}}$$

$${\sigma }_{\mathrm{U},i}^2$$

$${\text{pdf}}_X(x|o)\sim \mathcal{N}(x_{\mathrm{M},i}^{\mathrm{post}},{\sigma }_{\mathrm{M},\mathrm{post}}^2),$$

$${\text{pdf}}_O(o|x)={\text{pdf}}_O(o)\frac{{\text{pdf}}_X(x|o)}{{\text{pdf}}_X(x)},$$

$${\text{pdf}}_O(o|x)\sim \frac{\mathcal{N}(x_{\mathrm{M},i}^{\mathrm{post}},{\sigma }_{\mathrm{M},\mathrm{post}}^2)}{\mathcal{N}(x_{\mathrm{M},i}^{\mathrm{pri}},{\sigma }_{\mathrm{M},\mathrm{pri}}^2)},$$

$$x_{\mathrm{M},i}^{\mathrm{ext}}=E\{o|x_{\mathrm{M},i}^{\mathrm{pri}}\}={\sigma }_{\mathrm{M},\mathrm{ext}}^2\cdot \left(\frac{x_{\mathrm{M},i}^{\mathrm{post}}}{{\sigma }_{\mathrm{M},\mathrm{post}}^2}-\frac{x_{\mathrm{M},i}^{\mathrm{pri}}}{{\sigma }_{\mathrm{M},\mathrm{pri}}^2}\right)$$

$${\sigma }_{\mathrm{M},\mathrm{ext}}^2=\mathrm{Var}\{o|x_{\mathrm{M},i}^{\mathrm{pri}}\}={\left(\frac{1}{{\sigma }_{\mathrm{M},\mathrm{post}}^2}-\frac{1}{{\sigma }_{\mathrm{M},\mathrm{pri}}^2}\right)}^{-1},$$

$$z-g(z)=k_{E}e+n_E,$$

$$k_E=\frac{E_{ZE}\{(z-g(z))e\}}{{\sigma }_e^2}$$

$$k_E$$

$$=\frac{{\sigma }_e^2-E\{g(x+e)e\}}{{\sigma }_e^2}.$$

$$\underset{X,E}{E}\{g(x+e)\cdot e\}=\int \int g(x+e)e{\text{pdf}}_X(x){\text{pdf}}_E(e)\mathrm{d}e\mathrm{d}x$$

$$=\int {\text{pdf}}_X(x)\left(\int (z-x){\text{pdf}}_E(z-x)g(z)\mathrm{d}z\right)\mathrm{d}x$$

$$\stackrel{\text{mbox}def}{=}\int {\text{pdf}}_X(x)I_1(x)\mathrm{d}x$$

$${\sigma }_{\mathrm{S},\mathrm{post}}^2=\underset{Z}{E}\{{\sigma }_{\mathrm{S},\mathrm{post},i}^2(z)\}=\underset{X,E}{E}\{{\sigma }_{\mathrm{S},\mathrm{post},i}^2(x+e)\}$$

$$=\int \int {\sigma }_{\mathrm{S},\mathrm{post},i}^2(x+e){\text{pdf}}_X(x){\text{pdf}}_E(e)\mathrm{d}e\mathrm{d}x$$

$$=\int {\text{pdf}}_X(x)\left(\int {\text{pdf}}_E(z-x){\sigma }_{\mathrm{S},\mathrm{post},i}^2(z)\mathrm{d}z\right)\mathrm{d}x$$

$$\stackrel{\text{mbox}def}{=}\int {\text{pdf}}_X(x)I_2(x)\mathrm{d}x.$$

$$I_1(q)$$

$$I_2(q)$$

$$\displaystyle-\int\pdf_{E}(z-q)\ \bigg{(}\frac{\int x\pdf_{E}(z-x)\pdf_{X}(x)\,\mathrm{d}x}{f_{Z}(z)}\bigg{)}^{2}\,\mathrm{d}z\ .$$

$$\int f^{\prime }(z)g(z)\mathrm{d}z$$

Full text

I Introduction

In many communication scenarios, the transmitted vector {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}\in\R^{L\times 1} is sparse, i.e., only $s$ elements are non-zero.111Notation: $||\cdot||_{p}$ denotes the $\ell_{p}$ norm. {\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}_{(l,m)}=A_{l,m} is the element in the $l^{\mathrm{th}}$ row and $m^{\mathrm{th}}$ column of $\textstyle A$. {\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}^{\mathsf{T}} and {\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}^{-1} denote the transpose and the inverse of $\textstyle A$, respectively. \mathrm{\bf{diag}}({\mathchoice{\mbox{\boldmath\displaystyle a}}{\mbox{\boldmath\textstyle a}}{\mbox{\boldmath\scriptstyle a}}{\mbox{\boldmath\scriptscriptstyle a}}}) denotes a diagonal matrix of appropriate size with entries of the vector $\textstyle a$ as diagonal elements. \mathrm{\bf{diag}}({\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}) denotes a diagonal matrix with the same diagonal elements as $\textstyle A$. $\textstyle I$ is the identity matrix. $\mathcal{Q}_{\mathcal{C}}(\cdot)$: element-wise quantization w.r.t. a given alphabet $\mathcal{C}$. $\operatorname*{E}\{\cdot\}$: element-wise expectation. $\operatorname*{Var}\{\cdot\}$: Variance. $\mathcal{N}(m,v)$: Gaussian distribution with mean $m$ and variance $v$. $\pdf_{X}(x)$: probability density function of random variable $x$.

The receive vector {\mathchoice{\mbox{\boldmath\displaystyle y}}{\mbox{\boldmath\textstyle y}}{\mbox{\boldmath\scriptstyle y}}{\mbox{\boldmath\scriptscriptstyle y}}}\in\R^{K\times 1}, $s\ll K<L$, is corrupted by i.i.d. zero-mean Gaussian noise $\textstyle n$, with variance $\sigma_{n}^{2}$ per component. The channel can then be modeled by

[TABLE]

where {\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}\in\R^{K\times L} corresponds to the channel matrix in terms of communications engineering. Since $K<L$, a sparse vector has to be recovered from an underdetermined system of linear equations, a problem which is known as Compressed Sensing (CS) [1]. In many communication scenarios, however, the non-zero elements are not real-valued but drawn from a finite set $\mathcal{C}$. If the sparsity is fixed, the problem to be solved is given by ($\mathcal{C}_{0}\stackrel{{\scriptstyle\mbox{\scriptsize def}}}{{=}}\mathcal{C}\cup\{0\}$)

[TABLE]

There are many fields of digital communication in which this problem of estimating a discrete-valued sparse vector from an underdetermined system of linear equations is present, such as, e.g., sensor networks, where a fusion center with $K$ antennas has to reconstruct which of the $L$ low-activity sensors have currently been active, and which data has been transmitted by them [2]. Further applications are peak-to-average power reduction in orthogonal frequency-division multiplexing [3], the detection of pulse-width-modulated signals in radar applications [4], code-book excited linear prediction (CELP) source coding [5], and Compressed-Sensing-based cryptography [6].

There is a tremendously wide range of algorithms solving the standard continuous-valued CS problem, such as, amongst others, Orthogonal Matching Pursuit (OMP) [7], Iterative Hard Thresholding (IHT) [8], and Iterative Soft Thresholding (IST) [9]. Although the standard CS problem is non-convex due to its sparsity constraint, it can be relaxed to a $\ell_{1}$-based problem, which can efficiently be solved by the simplex algorithm or interior point methods [10].

In the case of discrete Compressed Sensing, however, additional information, i.e., the knowledge that the elements of the sparse vector are from a finite set, is available and has to be taken into account adequately. The estimation of a discrete-valued vector has combinatorial complexity in general, and hence discrete CS is non-convex, even if the constraint in problem (2) was relaxed to an $\ell_{1}$-based one, which could be solved by extensions of the simplex algorithm [10]. Unfortunately, these algorithms have a prohibitively high computational complexity.

Some algorithms for the solution of problem (2) have been proposed over the last few years. Besides the most obvious approach of a standard CS algorithm with subsequent quantizer [11], the quantization can be included inside OMP [13], which equals the so-called model-based Compressed Sensing [12] if it is applied to discrete CS. This algorithm has been further improved by the application of a method which preserves reliability information [13]. Another improved variant of OMP has been introduced in [14], where a minimum mean-squared error estimator has been applied.

Other algorithms for the CS problem are related to well-known channel decoding algorithms, e.g., the approximate message-passing (AMP) algorithm [15, 16], which is derived from the message-passing algorithm [17] and which can be easily adapted to cases (such as discrete CS) where information on the a-priori distribution of the sparse vector is available (Bayesian AMP, BAMP) [15]. Moreover, the knowledge from channel coding has also been used for the optimization of measurement matrices with adapted specialized recovery algorithms, cf., e.g., [18, 19]. The drawback of these approaches is that the restrictions on the measurement matrix limit the range of applications.

In [20, 21], an approach which is based on the turbo-principle has been proposed. It has been simplified, generalized, and adapted to the discrete setup in [22]. In this paper, this algorithm is further improved, thereby including knowledge from the field of digital communications into CS. The aim of this paper is to gain a profound understanding of the algorithm. An analysis of the approaches leads to intriguing insights into the algorithm, and especially into the important topic of bias compensation. Note that the results hold for standard CS and for discrete CS.

The paper is organized as follows. In Sec. II, the improved algorithm is introduced. The analysis is given in Sec. III. In Sec. IV, the performance of the algorithms is compared, followed by brief conclusions in Sec. V.

II Turbo Signal Recovery

In [20, 21], an iterative algorithm for the estimation of complex-valued sparse vectors has been presented. This algorithm, which has been denoted as Turbo Signal Recovery (TSR) (or Turbo Compressed Sensing), has been restricted to one type of measurement matrices in the original work. In [22], the algorithm has been generalized to a wider range of matrix construction, which allowed the comparison of TSR with other algorithms. Furthermore, the notation of the algorithm has been simplified. In this section, after a short explanation of this modified algorithm, an improved version is introduced.

II-A Approximate LMMSE TSR

In the generalized TSR algorithm, the measurement matrix is assumed to be given by {\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}={\mathchoice{\mbox{\boldmath\displaystyle U}}{\mbox{\boldmath\textstyle U}}{\mbox{\boldmath\scriptstyle U}}{\mbox{\boldmath\scriptscriptstyle U}}}{\mathchoice{\mbox{\boldmath\displaystyle C}}{\mbox{\boldmath\textstyle C}}{\mbox{\boldmath\scriptstyle C}}{\mbox{\boldmath\scriptscriptstyle C}}}, where $\textstyle U$ is a random part of a unitary matrix, and {\mathchoice{\mbox{\boldmath\displaystyle C}}{\mbox{\boldmath\textstyle C}}{\mbox{\boldmath\scriptstyle C}}{\mbox{\boldmath\scriptscriptstyle C}}}=\mathrm{\bf{diag}}([c_{1},\ldots,c_{L}]) is a scaling matrix. In the standard CS setup where the column vectors of $\textstyle A$ are assumed to be normalized to unit length, the scaling factors calculate to $c_{i}^{2}=1/\sum_{j=1}^{K}U_{j,i}^{2}$.

The sparse vector is estimated in an iterative fashion, where each iteration consists of two main parts: In the first part, all elements of the vector are estimated jointly by a linear estimator aiming to keep the Euclidean estimation error small, thereby ignoring the sparsity and the alphabet constraint. In the second part, the constraints which have been disregarded in the first part are taken into account, which leads to a non-linear estimator which generates so-called soft values [23]. The pseudocode of this algorithm is given in Alg. 1, Variant A, i.e., only the lines tagged by an “A” are active.

The authors of the original TSR algorithm denote the first step as linear minimum mean-square error (LMMSE) estimation [20]. In this step, the sparse vector is estimated by (cf. Alg. 1, Line 3)

[TABLE]

where {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{M}}^{\mathrm{pri}} is a prior estimate (from the previous step), $\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}$ is the variance of the estimation error of this prior estimate, and $\bar{c}^{2}=\frac{1}{L}\sum_{i=1}^{L}c_{i}^{2}$ is the average scaling factor. All variables of this MMSE-estimation-step are marked by the index “$\mathrm{M}$”.

In [22], it has been shown that with {\mathchoice{\mbox{\boldmath\displaystyle C}}{\mbox{\boldmath\textstyle C}}{\mbox{\boldmath\scriptstyle C}}{\mbox{\boldmath\scriptscriptstyle C}}}\approx\bar{c}{\mathchoice{\mbox{\boldmath\displaystyle I}}{\mbox{\boldmath\textstyle I}}{\mbox{\boldmath\scriptstyle I}}{\mbox{\boldmath\scriptscriptstyle I}}}, (3) can be approximated by

[TABLE]

Note that (when ignoring the scaling factor) this term corresponds to the first step in the well-known IHT algorithm [8], which can, on the one hand, be interpreted as one step of the gradient descent method, but, on the other hand, as correlation-based estimation, since $\textstyle x$ is estimated based on the correlation between the residual and the column vectors of $\textstyle A$. From a communications engineering point of view, it corresponds to the application of a matched filter. Hence, although claimed otherwise by the authors in [20], the first step is not an LMMSE estimation.

In the second step, soft values are calculated, which are the expected value of $x$ conditioned to the prior estimate from the first step and the a-priori distribution of $x$ (cf. Alg. 1, Line 8). Note that, in contrast to the joint estimation in the first step, this calculation is performed for each element of the sparse vector individually. Since this approach takes the a-priori distribution of $x$ into account, it depends on the alphabet; an adaptation to any alphabet is straightforward. This approach is also used in other algorithms for (discrete) CS, cf., e.g., [15, 22, 24, 13]. All variables of the second (soft-value calculating) step are indicated by the index “$\mathrm{S}$”.

This algorithm is denominated as TSR/Q in the following, where the trailing “Q” emphasizes the final quantization step which has to be performed in order to restrict the estimate to the discrete alphabet.

II-B Exact LMMSE TSR

After the introduction of TSR/Q in the previous section, an improved version of the algorithm is introduced in this section. To this end, the true linear MMSE estimate for $\textstyle x$ is derived. We assume the channel model (1), i.e., {\mathchoice{\mbox{\boldmath\displaystyle y}}{\mbox{\boldmath\textstyle y}}{\mbox{\boldmath\scriptstyle y}}{\mbox{\boldmath\scriptscriptstyle y}}}={\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}{\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}+{\mathchoice{\mbox{\boldmath\displaystyle n}}{\mbox{\boldmath\textstyle n}}{\mbox{\boldmath\scriptstyle n}}{\mbox{\boldmath\scriptscriptstyle n}}}, and want an estimate {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{M}}^{\mathrm{post}} of $\textstyle x$ that minimizes the expectation of the squared error given prior knowledge on $\textstyle x$. In general, the linear MMSE estimator is given by [25, 22]

[TABLE]

where $\textstyle d$ is the error vector if {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{M}}^{\mathrm{pri}}={\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}+{\mathchoice{\mbox{\boldmath\displaystyle d}}{\mbox{\boldmath\textstyle d}}{\mbox{\boldmath\scriptstyle d}}{\mbox{\boldmath\scriptscriptstyle d}}} is written as a noisy variant of $\textstyle x$, and {\mathchoice{\mbox{\boldmath\displaystyle\Phi}}{\mbox{\boldmath\textstyle\Phi}}{\mbox{\boldmath\scriptstyle\Phi}}{\mbox{\boldmath\scriptscriptstyle\Phi}}}_{dd}=\operatorname*{E}\{{\mathchoice{\mbox{\boldmath\displaystyle d}}{\mbox{\boldmath\textstyle d}}{\mbox{\boldmath\scriptstyle d}}{\mbox{\boldmath\scriptscriptstyle d}}}{\mathchoice{\mbox{\boldmath\displaystyle d}}{\mbox{\boldmath\textstyle d}}{\mbox{\boldmath\scriptstyle d}}{\mbox{\boldmath\scriptscriptstyle d}}}^{\mathsf{T}}\} is its correlation matrix. If we assume that the elements of $\textstyle d$ are uncorrelated with variance $\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}$, the estimation can be simplified to

[TABLE]

The calculation of the soft values in the second step is equal to the one in TSR.

This turbo-based algorithm, combining MMSE estimation and soft feedback, is denoted by TMS/Q. The pseudocode of this algorithm is given in Alg. 1, Variant B.

A comparison with the linear estimation in TSR/Q (cf. (4)) shows that TSR/Q can be seen as a simplified version of TMS/Q, valid for {\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}{\mathchoice{\mbox{\boldmath\displaystyle A}}{\mbox{\boldmath\textstyle A}}{\mbox{\boldmath\scriptstyle A}}{\mbox{\boldmath\scriptscriptstyle A}}}^{\mathsf{T}}\approx\bar{c}^{2}{\mathchoice{\mbox{\boldmath\displaystyle I}}{\mbox{\boldmath\textstyle I}}{\mbox{\boldmath\scriptstyle I}}{\mbox{\boldmath\scriptscriptstyle I}}}. Note that TMS/Q holds for all types of measurement matrices, the restriction that $\textstyle A$ is a (scaled) part of a unitary matrix, which was assumed in the original paper on TSR [20] and also for TSR/Q, is not required anymore.

III Discussion of the Extrinsics Calculation

The TMS algorithm is based on the turbo principle, which is well-known in channel coding [26]. The general idea is that two decoders (A and B in general, LMMSE estimator and soft value calculation here) exist, which alternately decode or estimate the signal and which exchange information about the current results. Each decoder takes into account the information from the other decoder (the so-called priors, i.e., {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{M}}^{\mathrm{pri}} & {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{S}}^{\mathrm{pri}}) and calculates a new estimate, the so-called posteriors ({\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{M}}^{\mathrm{post}} and {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{S}}^{\mathrm{post}}). In order not to hand back the information to the other decoder which has been received from it, this prior information has to be removed from the estimate (resulting in the so-called extrinsics, i.e., {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{M}}^{\mathrm{ext}} & {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{S}}^{\mathrm{ext}}) before passing it to the other decoder. The extrinsic values are calculated by (cf. Line 7 & 11, Alg. 1)

[TABLE]

with corresponding average variances

[TABLE]

This principle is shown in the upper part of Fig. 1.

In the following, both blocks of the algorithm are analyzed from a different point of view, leading to interesting insights.

III-A First Part

If discrete-valued signals are to be recovered, it is important that the diagonal elements of the end-to-end cascade for the estimation of $\textstyle x$ are equal to $1$, otherwise a bias is present. In the MMSE case, the cascade is given by

[TABLE]

with diagonal elements smaller than one. In order to compensate for the bias, the estimates have to be scaled by the inverse of the diagonal elements of $\textstyle K$. Given the biased estimate {\mathchoice{\mbox{\boldmath\displaystyle x}}{\mbox{\boldmath\textstyle x}}{\mbox{\boldmath\scriptstyle x}}{\mbox{\boldmath\scriptscriptstyle x}}}_{\mathrm{M}}^{\mathrm{pri}}, the $i^{\mathrm{th}}$ unbiased element $x_{\mathrm{U},i}$ is estimated by

[TABLE]

With the prior error variance $\sigma_{{\mathrm{M},\mathrm{pri}}}^{2}$, the biased a-posteriori error variance of the $i^{\mathrm{th}}$ element is given by [25]

[TABLE]

and the unbiased error variance $\sigma_{\mathrm{U},i}^{2}$ calculates to [22]

[TABLE]

Combining (8)–(10), it follows

[TABLE]

which is equal to the calculation of the extrinsic values (cf. Line 7, Alg. 1). Hence, the extrinsic calculation corresponds to unbiasing of the estimate, and thus, when using unbiased MMSE estimates, (inherently) extrinsic information is considered.

The unbiasing operation can also be interpreted from a third point of view. Given an observation $o$, the probability density function (pdf) of the biased a-posteriori estimate is given by

[TABLE]

which corresponds to the backward channel model, cf. Fig. 2, upper part. Note that in this model, as in any MMSE solution, the error is uncorrelated to the observation $o$. After the bias compensation, however, the error is uncorrelated to the unbiased estimate [27], which leads to the forward channel model which is shown in the lower part of Fig. 2. In this case, the density of interest is $\pdf_{O}(o|x)$. With Bayes’ theorem it holds

[TABLE]

where $\pdf_{O}(o)$ can be considered as a constant for a given $o$. If all variables are Gaussian distributed, it follows

[TABLE]

with the expected value and variance [28]

[TABLE]

which are again the equations used for the extrinsic calculation in TMS. Thus, the biased MMSE estimate is equal to the a-posteriori value, which corresponds to the backward channel model. The unbiased MMSE estimate is equal to the extrinsic value, and thus the forward channel model.

III-B Second Part

In the second part of TMS, soft values are calculated. Since the estimator is non-linear, the calculation of the bias is not as obvious as for the first step. However, every non-linear memoryless device can be written as a linear scaling plus an additive estimation error which is uncorrelated to the linear part [29]. In general, any prior estimate $z$ of the transmitted value $x$ (with $z=x_{\mathrm{S},i}^{\mathrm{pri}}$ in our case) can be written as noisy variant of $x$, i.e., $z=x+e$, where we assume the error to be Gaussian $e\sim\mathcal{N}(0,\sigma_{e}^{2})$, with $\sigma_{e}^{2}=\sigma_{\mathrm{S},\mathrm{pri}}^{2}$, and independent from $x$. The soft value of $z$ can be written as a function $x_{\mathrm{S},i}^{\mathrm{post}}=g(z)=\operatorname*{E}\{x_{i}|z\}$, $z=x_{\mathrm{S},i}^{\mathrm{pri}}$, which in turn is linearized as [29]

[TABLE]

where the scaling factor

[TABLE]

is chosen such that the error $n_{E}$ has minimum variance. Note that, in contrast to the first step where we estimate the signal $x$, the error $e$ is estimated, with which in turn $x$ can be calculated. The estimated error after the soft feedback is given by $x+e-g(x+e)$, and hence

[TABLE]

It can be shown via integration by parts that for any distribution of $x$, $\operatorname*{E}\{g(x+e)\,e\}=\operatorname*{E}\{g(x_{\mathrm{S},i}^{\mathrm{pri}})\,e\}=\sigma_{\mathrm{S},\mathrm{post}}^{2}=\operatorname*{E}\{x_{i}^{2}|x_{\mathrm{S},i}^{\mathrm{pri}}\}-(\operatorname*{E}\{x_{i}|x_{\mathrm{S},i}^{\mathrm{pri}}\})^{2}$ holds if the error is Gaussian distributed.222 For a detailed derivation, please see the Appendix.

Hence, taking $\sigma_{e}^{2}=\sigma_{\mathrm{S},\mathrm{pri}}^{2}$ into account, $k_{E}=\frac{\sigma_{\mathrm{S},\mathrm{pri}}^{2}-\sigma_{\mathrm{S},\mathrm{post}}^{2}}{\sigma_{\mathrm{S},\mathrm{pri}}^{2}}$, and with the resulting unbiasing factor $1/k_{E}$, the unbiased estimate calculates to $x_{\mathrm{S},i}^{\mathrm{ext}}=z-\frac{1}{k_{E}}\cdot(z-g(z))=x_{\mathrm{S},i}^{\mathrm{pri}}-\frac{\sigma_{\mathrm{S},\mathrm{pri}}^{2}}{\sigma_{\mathrm{S},\mathrm{pri}}^{2}-\sigma_{\mathrm{S},\mathrm{post}}^{2}}\cdot(x_{\mathrm{S},i}^{\mathrm{pri}}-x_{\mathrm{S},i}^{\mathrm{post}})$ and $\sigma_{\mathrm{S},\mathrm{ext}}^{2}=\operatorname*{E}\{(x-x_{\mathrm{S}}^{\mathrm{ext}})^{2}\}$, which, after straightforward modifications, results in the same unbiasing formulas as in the first step. Hence, as in the first step, the extrinsic calculation in TMS is equal to unbiasing, i.e., the change from the backward channel model to the forward one. The block diagram of TMS when it is interpreted from this unbiasing perspective is given in the lower part of Fig. 1

A comparison of both steps is given in Table I. While the first step performs a vectorwise linear estimation of $\textstyle x$ (implicitly assuming that $\textstyle x$ is Gaussian distributed), a non-linear symbolwise estimator for the error $\textstyle e$ is applied in the second step, thereby taking into account the sparsity and alphabet constraint.

IV Simulation Results

In this section, numerical results of the new algorithm are shown and compared to the ones of established algorithms. Two different channel matrices are used. First, $\textstyle A$ is constructed as random part of a random orthogonal matrix. Second, $\textstyle A$ is a random Gaussian matrix. In both cases, the columns are normalized to unit length, and $L=258$, $K=129$, $s=20$, $\mathcal{C}_{0}=\{-1,0,+1\}$. To ensure convergence, all algorithms perform $50$ iterations. The measure of interest when dealing with discrete values is the symbol error rate $\mathrm{SER}=\frac{1}{L}\sum_{i=1}^{L}\Pr\{\hat{x}_{i}\neq x_{i}\}$. The results are shown in Fig. 3. In the SVD-based case (upper figure), the new algorithm (yellow, dashed) performs as good as IMS/Q (red) [22]. TSR/Q (yellow, solid) is outperformed by $0.5\,\mathrm{dB}$, and the well-known BAMP/Q by $0.2\,\mathrm{dB}$. For comparison, the results of the standard algorithms for discrete CS (IHT/Q, blue, and ISFT/Q, green) are also shown.

For Gaussian sensing matrices, the new algorithm TMS/Q shows the best performance. Note that TSR/Q fails since the assumption that $\textstyle A$ is a part of an orthogonal matrix is not fulfilled. In general, the Gaussian matrices are less structured than the SVD-based, and only a smaller sparsity is tolerated to obtain the same performance. Note that the influence of the matrix is much larger for algorithms in which the matched filter is applied (IHT/Q, ISFT/Q) than for MMSE-estimation-based algorithms (TMS/Q, IMS/Q). However, also the aforementioned algorithms are able to find the correct result if the sparsity is small enough.

V Conclusion

In this paper, we have shown that the calculation of extrinsic information is equal to unbiasing, justifying the calculation performed in the second step of TSR and TMS. In addition, the TSR algorithm has been improved by the application of the LMMSE estimation, which leads to better numerical results than the established algorithms.