Perceptual Compressive Sensing based on Contrast Sensitivity Function: Can we avoid non-visible redundancies acquisition?

TL;DR

This paper introduces a perceptual compressive sensing method that leverages the contrast sensitivity function to skip acquiring information that is not visible to the human eye, reducing unnecessary data collection.

Contribution

It presents a new CS approach that selectively avoids non-visible redundancies, improving efficiency by integrating perceptual models.

Findings

Reduces data acquisition by excluding non-visible information

Demonstrates improved efficiency in compressive sensing

Validates approach with experimental results

Abstract

In this paper, we propose a novel CS approach in which the acquisition of non-visible information is also avoided.

Equations

$${\mathbb{P}}_{{\ell }_0}:\underset{\mathbf{s}\in {\mathbb{R}}^N}{\min }{\Vert \mathbf{s}\Vert }_0$$

$$\text{s.t.}\mathbf{y}=\underset{\mathbf{\theta }}{\underbrace{\mathbf{\Phi \Psi }}}\mathbf{s}$$

$${\mathbb{P}}_{{\ell }_1}:\underset{\mathbf{s}\in {\mathbb{R}}^N}{\min }{\Vert \mathbf{s}\Vert }_1$$

$$\text{s.t.}\mathbf{y}=\mathbf{\theta }\mathbf{s}$$

$${\mathbb{P}}_{RW{\ell }_1}:\underset{\mathbf{s}\in {\mathbb{R}}^N}{\min }\sum _{i=1}^N{w}_i\left|s_i\right|$$

$$\text{s.t.}\mathbf{y}=\mathbf{\theta }\mathbf{s}$$

$$w_i^{\left(l+1\right)}=\frac{1}{\left|s_i^{\left(l\right)}\right|+\epsilon }$$

$$H\left(f_{i,j}\right)=2.6\left(0.0192+0.114f_{i,j}\right)e^{-{\left(0.114f_{i,j}\right)}^{1.1}}$$

$$f_{i,j}=\frac{1}{2N}\sqrt{{\left(\frac{i}{{\theta }_x}\right)}^2+{\left(\frac{j}{{\theta }_y}\right)}^2}\left(\text{cycles/degree}\right)$$

$${\theta }_x=2\arctan \left(\frac{{\Lambda }_x}{2D}\right),{\theta }_y=2\arctan \left(\frac{{\Lambda }_y}{2D}\right)$$

$${\theta }_x={\theta }_y=2\arctan \left(\frac{1}{2R_{vd}Pi{c}_h}\right)$$

$${\mathbb{P}}_{RW{\ell }_1}:\underset{\mathbf{s}\in {\mathbb{R}}^N}{\min }{\Vert \mathbf{Ws}\Vert }_1$$

$$\text{s.t.}\mathbf{y}=\mathbf{\theta }\mathbf{s}$$

$${\mathbb{P}}_{RW{\ell }_1}:\underset{\mathbf{z}\in {\mathbb{R}}^N}{\min }{\Vert \mathbf{z}\Vert }_1$$

$$\text{s.t.}\mathbf{y}=\mathbf{\theta }{\mathbf{W}}^{-1}\mathbf{z}$$

$${\mathbf{W}}^{-1}=\text{Diag}\left(\text{vec}\left(\mathbf{H}\right)\right)$$

Full text

I Introduction

Compression is a fundamental part of the signal processing. Since there exist a lot of redundancies when we sample a signal using Nyquist rate, therefore, it needs to be compressed. It is known that two general types of redundancies exist in an image: statistical and perceptual redundancies. Statistical redundancies could be eliminated using popular approaches like JPEG which incorporates the transform coding and variable length coding [1]. Perceptual redundancies depend on the human visual system (HVS). The sensitivity of the human eye is different for each spatial frequency which is defined as a number of sine or square wave gratings per visual angle [2]. There are a lot of reasearch in this direction to provide a mathematical model of human perception. In fact, the human eye is sensitive to distortion above a threshold level. Contrast sensitivity function (CSF) is proposed to model this kind of sensitivity [3]. It is also shown that the HVS behavior is like a band pass filter for luminance frequency. Therefore, perceptual redundancies can be removed by assigning the weights to the transform coding coefficients before quantization using CSF. In addition to CSF, there are also other effects which are image dependent like luminance masking and contrast masking [4].

It is well known that Compressive sensing (CS) efficiently acquire the most important part of a signal by sampling below the Nyquist rate [5, 6]. In fact, it alleviates the existed statistical redundancies. Recently, there has also been a lot of interest to adapt the weighted compressive sensing (WCS) to enhance the sparsity [7, 8, 9]. Iteratively reweighted least squares minimization (IRLS) approach of [7] has been proposed to benefit from regularization. The authors in [8] also proposed another IRLS approach to enhance the sparsity. In contrast to IRLS approaches, in [9], Candes et al. have proposed to design the weights based on the $\ell_{1}$ norm using log-sum penalty. Since no certain guarantees have been made for algorithm’s success in [9] and the error bound is not provided, the authors in [10], obtained a theoretical justification of minimizing log-sum penalty. However, these methods are iterative and computationally complex than the unweighted one. Therefore, the question would be how we could design the non-iterative weighting matrix for CS?

There are a few works that have attempted to design the weighting coefficients using HVS. In [11], the authors proposed foveated compressed sensing approach which utilizes HVS. This method is used for image compression and not for image acquisition since it needs foveal information. Moreover, in [12], for two-dimensional discrete cosine transform (DCT) the weighting coefficients for measurement matrix are derived from the standard JPEG quantization table by taking the inverse of the table entries and adjusting their amplitudes to a proper range. Similarly, in [13], we have proposed to design the weighting coefficients for enhancing sparsity based on the standard JPEG quantization table for a video sequence.

Motivated by the fact that the human eye is not much sensitive to the high frequency details, we could give an importance to each frequency in signal reconstruction, such that the high frequencies have a small effect on the performance. Hence, we could expect better visuality performance from the proposed scheme compared to other conventional weighted methods. One benefit that could be expected for this approach is the weighting coefficients are independent of input images which make it non-iterative and fast. In summary, different from the mentioned weighted approaches, here we propose to benefit from contrast sensitivity function to design weighting coefficients with the aim of enhancing sparsity.

The rest of the paper will be organized as follows. Section II provides a brief overview on CS and WCS. Section III presents the proposed method. Simulation results and discussions are given in section IV. Finally, section V concludes the paper and some future directions are presented.

II Background

II-A Compressive Sensing (CS)

Consider a signal, $\mathbf{x}\in{{\mathbb{R}}^{N\times 1}}$ which possesses sparse representation in a domain ${\mathbf{\Psi}}\in{\mathbb{R}^{N\times N}}$, i.e. ${\mathbf{x}}={\mathbf{\Psi s}}$ where ${\mathbf{s}}\in{\mathbb{R}^{N}}$ and ${\left\|{\mathbf{s}}\right\|_{0}}=k$ (${\left\|.\right\|_{0}}$ stands fo the $\ell_{0}$ norm). According to CS theory, this signal can be recovered uniquely from linear measurements ${\mathbf{y}}\in{\mathbb{R}^{M}}$: ${\mathbf{y}}={\mathbf{\Phi x}}={\mathbf{\Phi\Psi s}}$ where ${\mathbf{\Phi}}\in{\mathbb{R}^{M\times N}}$ is a full-rank sensing matrix with $\left(M<N\right)$. Gaussian measurement matrices satisfy restricted isometry property (RIP) constraint with high probability if the number of measurements was, at least, equal to $M=\mathcal{O}\left({k\log\left({{N\mathord{\left/{\vphantom{nk}}\right.\kern-1.2pt}k}}\right)}\right)$ [6]. Therefore, the signal can be uniquely recovered by solving the following minimization problem

[TABLE]

Although solving problem (II-A) gives the exact solution, but unfortunately, it is NP-hard requiring an exhaustive search over all $\binom{N}{k}$ possible solutions. There are some classical sparse approximation methods which find the approximate solution like Matching Pursuit (MP) [14] and Orthogonal MP (OMP) [15], etc. However, these greedy algorithms are fast, but they did not have worthy performance. One alternative method to avoid NP-hardness of the zero norm minimization is to replace zero norm with smallest convex $\ell_{p}$ norm which is $\ell_{1}$ norm. Hence, the problem (II-A) transformed into following $\ell_{1}$ minimization problem:

[TABLE]

It is proved that this optimization problem can exactly recover the $k$-sparse signals and closely approximate the compressible signals with high probability [6, 16]. However the computational complexity of this approach is about $\mathcal{O}\left({{N^{3}}}\right)$ and could be kind of slow for large size signals. Hence there are some approaches trying to find a better function for approximating the zero norm. In [17], the authors have tried to directly minimize the smoothed version of zero norm which is called SL0 and is 2 or 3 orders of magnitude faster than $\ell_{1}$ minimization. In SL0 approach, the exponential smoothed version of the zero norm is considered.

II-B Weighted Compressive Sensing (WCS)

The authors in [9] have tried to replace zero norm with log-sum penalty function which resulted in iterative reweighted $\ell_{1}$-minimization. It has shown that this approach reconstructs the exact $k$-sparse signal ($\mathbf{\Psi}=\mathbf{I}$) with a lower number of measurements than $\ell_{1}$-minimization algorithm, however, the computational complexity is increased because of the iterative reweighting structure. The reweighted ${\ell_{1}}$-minimization approach is defined as:

[TABLE]

where ${{w}_{i}}>0$ denotes the weights at index $i$. To obtain the solution with the same sparsity structure of the original signal, ${w}_{i}$ should have small values on the nonzero locations of signal and significantly larger values elsewhere. Since we haven’t the prior information about the location and amplitude of the non-zero elements of the sparse signal, selecting weights is done by iterative manner. Common approaches for iteratively computing weights are based on recalculating weights at every iteration using the solution of (II-B) at the previous iteration. Suppose ${s}_{i}^{\left(l\right)}$ denotes the $i$’th element of the solution of (II-B) for a given set of weights in $l$’th iteration. In the next iteration, the weights are updated as [9]:

[TABLE]

where $i=1,2,...,{N}$, and $\varepsilon$ is a positive parameter in order to provide stability especially when ${{s}_{i}^{\left(l\right)}}$ is zero valued. It should be noted that the iterative reweighted $\ell_{1}$-minimization algorithm has better performance for exactly $k$-sparse signals. However, the improvement of this approach decreases for compressible signals.

III Proposed approach

Since the RWL1 approach [9] is iterative and it has a higher computational complexity compared to unweighted CS, in this research, we try to answer this question: how we could design the non-iterative and image independent weighting matrix of CS? Motivated by the human visual system (HVS) which state that the sensitivity of human eye is not same for different frequencies, we could give an importance to each frequency in signal reconstruction. Hence, we could expect better visuality performance from the proposed scheme compared to other state-of-the-art WCS methods. One benefit that could be expected for this approach is that the weighting coefficients are image independent which makes it non-iterative and fast.

To design a perceptual WCS scheme, first, we should know how humans see the world. There are a lot of research in this direction to provide a mathematical model of human perception. The human eye is sensitive to the spatial frequency which is defined as a number of sine or square wave gratings per visual angle. The function that model this effect is called contrast sensitivity function (CSF). The following model for CSF is originally proposed by Mannos and Sakrison [3] which is plotted in Fig. 1.

[TABLE]

where ${f_{i,j}}$ stands for the spatial frequency which is defined in the DCT domain as follows:

[TABLE]

where $i,j=0,\cdots,N-1$. Here, $\theta_{x}$, $\theta_{y}$ denote the visual angles in the $x$ and $y$ direction and are defined as follows:

[TABLE]

where $D$ is the viewing distance and $\Lambda_{x}$, $\Lambda_{y}$ denote the display width/length of a pixel on the monitor. Let us define $R_{vd}$ as the ratio of viewing distance to picture height. According to the international standard ITU-R BT.500-11 [18] (Methodology for the subjective assessment of the quality of television pictures), this ratio normally is between 3 to 6 depending on the picture size. It is also known that for most of the displays, pixel aspect ratio (PAR) is equal to 1. Therefore, we can simplify (7) as follows:

[TABLE]

where $Pic_{h}$ is stand for picture height. Now, let us explain how we assign the weights perceptually. It is evident that Eq. (II-B) can be written as follows:

[TABLE]

where $\mathbf{W}$ refers to a diagonal matrix with $\left\{{w_{i}}\right\}_{i=1}^{N}$ as a diagonal elements. Assume that $\mathbf{z}=\mathbf{Ws}$. Therefore, (III) is equivalent to

[TABLE]

Note that it is proved in [9] that the weighting coefficients are inversely proportional to the value of the signal, since the larger values of the signal should be less penalized. Considering this issue, we propose to use (5) as an inverse of weight coefficient for each element in the DCT domain. Mathematically,

[TABLE]

where the $\left(i,j\right)$’th entry of the matrix $\mathbf{H}$ is equal to $H\left(f_{i,j}\right)$. As a last step of our proposed method, we assign the maximum value for the inverse of weight to the zero frequency, i.e. DC component since it has the highest value in the DCT domain. To better understand the way of weight allocation to each frequency in the DCT domain, in Fig. 2, we have visualized the distribution of the CSF matrix ($\mathbf{H}$) in our proposed scheme and we have also compared it with the one in [13] which is based on the modified JPEG quantization matrix. As it can be seen, the CSF matrix is symmetric while the modified JPEG quantization matrix in [13] is not symmetric.

IV Experiments

In this section, we provide some simulations to validate the effectiveness of our proposed algorithm. Fig. 3 shows six popular test images which are used in this paper, i.e. the Mondrian, Lenna, Cameraman, Tile Roof, Barbara and Clock test images with different sizes. Since the Block-based CS is used to make our approach practical, the two-dimensional image is divided into $B\times B$ blocks and then each block is sampled with an ordinary random Gaussian matrix. Also, the 2D-DCT sparsifying basis is considered in our simulations. We employ blocks of size $B=16$ for our simulations. Also, the SparseLab software [19] is used for solving the $\ell_{1}$-minimization problems. Finally, we have measured the accuracy of the reconstructed signal using the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [20].

Fig. 4 and 5 compare the PSNR and SSIM of our proposed scheme with the conventional CS and the approach of [13] respectively. Here, we do not include the results of RWL1 [9] and IRLS [7, 8] since we have already shown in [13] that their performance is not competitive with the unweighted approach for compressible signals. From Fig. 4 and 5, it can be seen that our method is competitive with [13] which is based on the JPEG standard quantization matrix. We also observed that when the available number of measurements is low, then our approach has the best performance among the other methods. Comparing the test images, we observe that the reconstruction performance is greatly improved for images that have more perceptual redundancies like the Lenna, Cameraman, and Clock test images. Fig. 6 verified that the proposed CSF-based perceptual CS algorithm has better visual result in comparison to other methods. These results are plotted with $M=0.2N$. Our approach has better compressibility due to avoiding acquisition of perceptual redundancies. Therefore when the number of measurements is fixed, the performance of our method becomes better compared to other methods. It should be noted that we have also observed that by increasing the block size, the performance of all schemes are improved since the blocking artifact is reduced.

V Conclusion

In this paper, we propose a novel weighted compressive sensing approach based on contrast sensitivity function. The aim of conventional CS approaches is to avoid acquisition of statistical redundancies existed in the signal. Here, we design some perceptual weights to avoid acquisition of non-visible redundancies. In fact, these weights are used to enhance the sparsity. Moreover, in contrast to other weighted CS approaches, our scheme is non-iterative and fast. We also present some simulations to verify the effectiveness of our proposed scheme compared to other state-of-the-art methods. We use contrast sensitivity function to design weights since the aim of this paper is to design non-iterative weights. As a future work, we can design perceptual weights based on CSF in the wavelet domain. We can also benefit from luminance adaption and contrast masking to design iterative approaches which avoid non-visible redundancies and depends on the image content.