Boosting Dictionary Learning with Error Codes

TL;DR

This paper introduces a feedback-enhanced dictionary learning method that uses error codes to improve convergence and performance, especially with random initializations and high-dimensional data.

Contribution

It proposes a novel feedback process that leverages error codes to refine sparse representations, enhancing convergence and robustness in dictionary learning.

Findings

Faster convergence in high-dimensional settings

Better final dictionary states with random initializations

Scales well with increased sparsity constraints

Abstract

In conventional sparse representations based dictionary learning algorithms, initial dictionaries are generally assumed to be proper representatives of the system at hand. However, this may not be the case, especially in some systems restricted to random initializations. Therefore, a supposedly optimal state-update based on such an improper model might lead to undesired effects that will be conveyed to successive iterations. In this paper, we propose a dictionary learning method which includes a general feedback process that codes the intermediate error left over from a less intensive initial learning attempt, and then adjusts sparse codes accordingly. Experimental observations show that such an additional step vastly improves rates of convergence in high-dimensional cases, also results in better converged states in the case of random initializations. Improvements also scale up with more lenient sparsity constraints.

Tables

		$k=2$	$k=5$	$k=10$	$k=20$
Rand. 1.
MOD		26.33	32.08	36.82	40.43
EcMOD		26.91	33.90	37.87	41.72
EcMOD+		26.89	33.90	38.10	41.86
Rand. 2.
MOD		26.61	32.20	36.89	40.68
EcMOD		27.92	33.36	36.88	40.25
EcMOD+		26.73	33.87	38.08	41.83
DCT Init.
MOD		26.22	33.02	38.40	42.82
EcMOD		26.95	34.05	38.06	41.98
EcMOD+		26.94	33.96	38.19	42.07

Equations

$$\begin{array}{r}\underset{\mathbf{D},\{{\mathbf{x}}_i\}}{\mathrm{argmin}}\sum _{i=1}^M\Vert {\mathbf{y}}_i-\mathbf{D}{\mathbf{x}}_i{\Vert }_2^2\text{subject to}\Vert {\mathbf{x}}_i{\Vert }_0\le k.\end{array}$$

$$\begin{array}{r}\underset{\{{\mathbf{x}}_i\}}{\mathrm{argmin}}\sum _{i=1}^M\Vert {\mathbf{y}}_i-\mathbf{D}{\mathbf{x}}_i{\Vert }_2^2\text{subject to}\Vert {\mathbf{x}}_i{\Vert }_0\le k,\end{array}$$

$$\begin{array}{r}\underset{\mathbf{D}}{\mathrm{argmin}}\sum _{i=1}^M\Vert {\mathbf{y}}_i-\mathbf{D}{\mathbf{x}}_i^{\ast }{\Vert }_2^2.\end{array}$$

$$\begin{array}{r}\underset{\mathbf{D},\{{\mathbf{a}}_i\}}{\mathrm{argmin}}\sum _{i=1}^M\Vert {\mathbf{y}}_i-\mathbf{D}{\mathbf{a}}_i{\Vert }_2^2\text{subject to}\Vert {\mathbf{a}}_i{\Vert }_0\le m\end{array}$$

$$\underset{\{{\mathbf{b}}_i\}}{\mathrm{argmin}}\sum _{i=1}^M\Vert {\mathbf{e}}_i-{\mathbf{D}}^{\ast }{\mathbf{b}}_i{\Vert }_2^2\text{s.t.}\Vert {\mathbf{b}}_i{\Vert }_0\le k-m.$$

$$\begin{array}{r}\underset{\mathbf{D}}{\mathrm{argmin}}\sum _{i=1}^M\Vert {\mathbf{y}}_i-\mathbf{D}({\mathbf{a}}_i^{\ast }+{\mathbf{b}}_i^{\ast }){\Vert }_2^2.\end{array}$$

Full text

Index Terms—  Dictionary learning, sparse coding, error coding, boosting.

1 INTRODUCTION

One of the well-known properties of the Fourier transform and the wavelet transform is to exploit certain structures of the input signal and to represent these structures in a compact (or sparse) manner. Sparse representations have thus become an active research topic while providing good performance in a large diversity of signal and image processing applications [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].

Sparse representations consist in representing most or all information contained in a signal with a (linear) combination of a small number of atoms carefully chosen from an over-complete (redundant) basis. This basis is generally referred to as a dictionary. Such a dictionary is a collection of atoms whose number is much larger than the dimension of the signal space. Any signal then admits an infinite number of sparse representations and the sparsest such representation happens to have interesting properties for a number of signal and image processing tasks.

The most crucial question in sparse representations is the choice of the dictionary $\bf D$. One can realize a variety of pre-defined sets of waveforms, such as wavelets [11], curvelets [12], contourlets [13], shearlets [14], bandelets [15]. However, both the sparsity and the quality of the representation depend on how well the used dictionary is adapted to the data at hand. The problem of dictionary learning, or even simply finding adaptive ways to construct or to select relevant dictionaries, for sparse representations —that goes far beyond using a few bases (like DCT, DFT, wavelets, or the others)— has therefore become a key issue for further progress in this area.

Various dictionary learning algorithms have been proposed in the literature, e.g., [16, 17, 18, 19, 20, 21, 22, 23]. Most of these methods focus on $\ell_{0}$ or $\ell_{1}$ norm sparsity measures, potentially leading to simple formulations, hence to efficient techniques in practice. Method of Optimal Directions (MOD) [16] is one of those successful methods of non-parametric dictionary learning. MOD builds upon the K-means process alternating between a sparse coding step and a least-squares optimization based update step of the dictionary. Although MOD is effective for low dimensional cases, because of pseudo-inverse computation, it becomes intractable in high dimensional problems. K–SVD [17] is a similar algorithm, alternating between sparse coding and dictionary update. The difference in dictionary update step is that, instead of updating $\bf D$ as a whole, K–SVD updates single dictionary atoms and corresponding sparse codes, using the rank-1 approximation by minimizing the approximation error when the corresponding atom has been isolated from the dictionary. Therefore, these iterative updates affect following ones, presumably accelerating convergence.

It is important to note that, both in MOD and K–SVD, sparse coding with $\ell_{0}$ norm constraint is assumed. Coding methods for such constraint include matching pursuit (MP) [24] and orthogonal MP (OMP) [25] algorithms in general. However, there is no theoretical restriction to use $\ell_{1}$ norm constraint instead. Therefore, in essence, both MOD and K–SVD define specific dictionary update procedures resulting in non-structural learned dictionaries, rather than defining the type of sparse coding constraint. On the other hand, certain dictionary learning algorithms are specifically based on coding with $\ell_{1}$ norm constraint. In this context, coefficient learning will be a more appropriate term than sparse coding when referring to calculation of sparse representation coefficients. Basis pursuit (BP)[26], LARS [27], FOCUSS [28] can be listed as examples. Gradient descent or Lagrange dual methods can then be chained as a dictionary update step [29]. These alternative algorithms are proved to be more efficient especially in terms of convergence rates.

Accomplishments of such $\ell_{1}$ norm based algorithms over conventional $\ell_{0}$ ones are apparent. However, as noted before, $\ell_{1}$ norm is a relaxation on the original problem, so it does not guarantee an optimal solution in general. Such approach is especially problematic in low-dimensional cases. On the other hand, because of its non-convexity properties, $\ell_{0}$ norm approach leads to low convergence rates, being feasible only in low-dimensional cases.

A usual deficiency in sparse coding step is that, algorithms listed above assume proper dictionaries at each iteration. This is indeed very problematic, especially at initial stages of the learning process. In many situations, initial dictionary will not be a good representative of the optimal one. Therefore, “optimal” coding done with such a dictionary, as targeted by both $\ell_{0}$ and $\ell_{1}$ norm coding schemes, will most likely result in sparse codes, which also are not good representatives of optimal state. As a result, the next dictionary will adopt this undesired property to a certain extent and convey it to successive learning steps. In this paper, we propose a generic modification to sparse coding or the coefficient learning step, with an error feedback process by coding an intermediate error and adjusting sparse codes accordingly in a less intensive learning attempt, hence leading to a faster convergence when compared to the conventional approaches.

This paper is organized as follows. Section 2 describes the details of the proposed dictionary learning algorithm. Section 3 demonstrates the experimental setup and the obtained results. Finally, Section 4 concludes this paper.

2 Dictionary Learning with Error Codes

2.1 Dictionary Learning

The problem of dictionary learning for sparse representations can be formulated as a non-convex optimization problem in the form of the equation as below,

[TABLE]

Here the set $\{{\bf y}_{i}\},i=1...M,$ represents training samples, $\bf D$ is the dictionary to be learned and ${\bf x}_{i}$ is the sparse representation of the sample ${\bf y}_{i}$, $\forall i$. Parameter $k$ defines the sparsity constraint allowed for sparse coding during the learning process. Since Eqn. (1) poses a non-convex optimization problem through an $\ell_{0}$ norm constraint, solving this problem is thought to be NP-hard [30]. However, it is shown that for many high dimensional cases $\ell_{1}$ norm constraint is enough to ensure sparsest solution [31]. Note that $\ell_{1}$ norm constraint also turns the problem into a convex optimization one, which can then be solved via regular convex optimization tools. However, this should still be regarded as an approximation to the original problem.

In essence, traditional algorithms split Eqn. (1) into two approximate subproblems and alternate between these two simpler but convex optimization problems, namely sparse coding and dictionary update steps, to find a solution as follows,

[TABLE]

[TABLE]

Eqn. (2) corresponds to the sparse coding step where the dictionary $\bf D$ is assumed to be fixed. This subproblem can easily be solved with any pursuit algorithm. Eqn. (3) defines the dictionary update step that is performed with $\{{\bf x}^{*}_{i}\}$, i.e., sparse codes acquired from sparse coding. One direct way to solve this subproblem is the least-squares optimization as in MOD, such that ${\bf D}^{*}={\bf Y}{\bf X}^{+}$ where ${\bf Y}=\{{\bf y}_{i}\}_{1}^{M}$ and ${\bf X}=\{{\bf x}^{*}_{i}\}_{1}^{M}$ represent the training samples matrix and sparse codes matrix, respectively. ${\bf X}^{+}$ is the Moore-Penrose pseudo-inverse of $\bf X$. A full iteration is completed alternating between sparse coding and dictionary update, and this procedure is repeated until convergence.

2.2 Introducing Error Feedback

We propose a formulation that incorporates an intermediate error into the learning process. In the first stage, a regular sparse coding and dictionary update procedure is performed but with a sparsity level $m<k$ by solving Eqn. (4).

[TABLE]

Let us now denote $\{{\bf a}^{*}_{i}\}$ and ${\bf D}^{*}$ as the resulting sparse codes and the dictionary, respectively. The second stage involves sparse coding the approximation error ${\bf e}_{i}={\bf y}_{i}-{\bf D}^{*}{\bf a}^{*}_{i}$, $\forall i$, as

[TABLE]

After acquiring $\{{{\bf b}^{*}_{i}}\}$ in Eqn. (5), current-state sparse codes can further be updated as ${{\bf a}^{*}_{i}+{\bf b}^{*}_{i}}$. This step basically corresponds to some sort of feedback logic, where the first approximation is tested and then its deviation is sparse coded to be incorporated into actual codes. Note here that the original sparsity constraint still holds since ${\|{\bf a}^{*}_{i}+{\bf b}^{*}_{i}\|_{0}\leq k}$. In the last stage, a final dictionary update is performed as in Eqn. (6) and an iteration is completed,

[TABLE]

3 EXPERIMENTAL RESULTS

Two variants of the proposed scheme have been tested experimentally, namely EcMOD in Algorithm 1 and EcMOD+ in Algorithm 2. EcMOD includes the methodology that is defined in Section 2, and EcMOD+ includes a regular least-squares dictionary update (MOD) at the end of each iteration. OMP is used for sparse coding.

Two experimental setups have been performed, corresponding to low and high dimensional cases respectively. In the first setup, $8\times 8$ distinct patches were extracted from the Barbara image of size $512\times 512$, resulting in $4096$ image patches. Dictionary size was accordingly chosen as $64\times 256$. Sparsity constraint $k$ and additional sparsity parameter $m$ were chosen as $8$ and $4$ respectively, so $m$ being equal to $k/2$. Results corresponding to this setup are presented in Figure 1, Figure 2 and Table 1.

In error coded schemes, as a consequence of not directly coding with sparsity $k$, final codes ${\bf a}^{*}_{i}+{\bf b}^{*}_{i}$ may not necessarily be optimal for $k$. However, as there are two coding steps with lesser sparsity constraints and a summation, codes have a higher chance of being optimal for most of the sparsity levels less than $k$. As a result, converged dictionary is a better representative of such sparsity levels, as observable in the results in Table 1. Error coded schemes consistently perform better in sparser cases.

Not targeting a sparsity level $k$ directly leads to a possibility of converged dictionary to be suboptimal for that given $k$. However, this drawback can be worked around by chaining a conventional step that targets an exact sparsity level $k$. Referred to as EcMOD+ algorithm, experiments with this further modified method show that, such architecture possesses optimality for $k$ sparsity and also better performance for the cases where sparsity level is less than $k$. This phenomenon is apparent in Figure 1, where learning and testing k chosen both as 8. Performance of EcMOD is not consistent as it performs much like MOD in second random initialization case. Whereas, EcMOD+ consistently performs well.

In a more extensive manner, Figure 2 compares the performance of MOD and EcMOD+ in the case of ten different randomly initialized dictionaries. DCT convergence is supplied as a baseline. This figure represents superiority of this error coding scheme over conventional coding in the case of random initializations. ”Optimal” coding with an improper random dictionary within initial stages hamper the final convergence state as observable in the case of MOD. Although not targeting optimal codes, error coded scheme EcMOD+ converges to DCT result in all 10 random initialization cases. This is possible because, in each step from the beginning, dictionary passes through a less intensive validation, in the expense of acquiring optimal codes. There is no total superiority in DCT initialization case as seen in Table 1. However, superiority can be achieved with more complex error coding schemes.

Finally, regarding overall sparsity levels within error codes and its evolution during learning process, the proposed method presents interesting trends. In conventional sparse coding (targeting the sparsity $k$), as approximation threshold is kept very strict in general, sparse codes end up with using all $k$ supports. Therefore, in methods such as MOD and K–SVD, codes consistently have $k$ supports even starting from the initial iteration. In the proposed method, during error coding, selection of previously selected supports is frequent. This is especially observable during initial iterations. Near maximum support counts are gradually reached as the system converges, but not necessarily reaching exact maximum.

In the second set of experiments, EcMOD+ scheme has been tested with all possible $255025$ image patches extracted with a full coverage of sliding window algorithm with a window size of $8\times 8$. Combinations of sparsity of $4$, $8$, $16$, and $32$ against small and large dictionaries were tested. DCT was used as initial dictionary in all cases. Note that, in DCT initialization cases, there is only an advantage of faster convergence rates but not of better converged states, at least for this error coding scheme. Superior converged states with more complex schemes have been achieved, but they are omitted here because of the space limitation.

Experimental results with the second setup are summarized in Figure 3 and Figure 4. In Figure 3, performance ratios were calculated relative to the MOD algorithm in terms of mean-squared error to estimate a performance gain factor for each sparsity level, approximately at fifth equivalent iteration, for an approximate convergence rate analysis. Gains for large dictionary in the case of learning with lenient sparsity levels are more striking, but stricter sparsity constraints cause substandard performance. Overall, the performance in this case is promising as it signals to scaling with input size. Smaller sized dictionary safely performs above standard. Figure 4 depicts the convergence plot for dictionary size 64 x 256 and k as 8 for both learning and testing. Significant gain in convergence rate is observable with the error coded scheme in this high-dimensional setup. Note that, mean-squared error is given as measure since sliding window patches were used. Finally, Figure 5 compares atoms that lie within similar frequency domains, learnt with MOD and EcMOD+. Note here the well-defined structure of EcMOD+.

4 CONCLUSION

MOD, by itself is a greedy algorithm that targets optimality one task at a time. Tasks are considered as isolated from each other, even within the same iteration. This results in MOD being a rather short-sighted method which fails at tasks that require a broader perspective of the system.

In this paper, we presented a method in which sparse coding and dictionary update steps are intertwined through intermediate error codes. Note that there could be other ways to accomplish this. Another way could be to add sparse codes of two successive iterations and perform a dictionary update based on this accumulated code, without even introducing error codes. As an analogy, MOD can be considered as a single-step numerical method, where as the example given would be a multi-step one. Our method can be considered as a multi-step approach that utilizes a half-step.

To summarize, our framework is generic enough to be included in many forms of learning-based approaches. In essence, our scheme includes an initial attempt of learning with less computational and spatial requirement than originally allocated. This corresponds to a single iteration of MOD performed with $m<k$. In this way, a feedback can be acquired that reflects the congruence of the model and the data at hand, so that current state can properly be adjusted before the final model update, which consumes the remaining resources.

This approach will be most beneficial for systems that are restricted to random initializations (as apparent in Figure 1, 2 and Table 1). A random initial model is most likely to be an improper representative of a specific system. Therefore, an update based on this model, no matter how intensive it is, will result in an undesired state. In fact, an optimal update based on this improper model could be more impairing than a suboptimal one in this regard.

As a concluding remark, readers should bear in mind that this work is based on a pragmatic perspective. Although satisfactory improvement has been observed, a more rigorous theoretical approach can lead to certain variations built on top of this framework that will be far more fruitful.