Face Recognition using Multi-Modal Low-Rank Dictionary Learning

TL;DR

This paper introduces a multi-modal low-rank dictionary learning approach for face recognition that effectively handles occlusion and illumination variations by fusing raw pixel data with illumination-invariant features.

Contribution

It presents a novel multi-modal structured low-rank dictionary learning method that improves robustness and discriminability in face recognition under challenging conditions.

Findings

Outperforms existing methods on various datasets.

Robust to severe illumination changes and occlusion.

Effective with limited training samples.

Abstract

Face recognition has been widely studied due to its importance in different applications; however, most of the proposed methods fail when face images are occluded or captured under illumination and pose variations. Recently several low-rank dictionary learning methods have been proposed and achieved promising results for noisy observations. While these methods are mostly developed for single-modality scenarios, recent studies demonstrated the advantages of feature fusion from multiple inputs. We propose a multi-modal structured low-rank dictionary learning method for robust face recognition, using raw pixels of face images and their illumination invariant representation. The proposed method learns robust and discriminative representations from contaminated face images, even if there are few training samples with large intra-class variations. Extensive experiments on different datasets validate the superior performance and robustness of our method to severe illumination variations and occlusion.

Tables

Table 1. Table 1 : Recognition rates (%) on AR dataset

Method	Sunglasses	Scarf	Mixed	Misc.
MLDL [9]	90.51	91.51	91.32	76.33
UMD2L [8]	88.26	87.40	88.30	71.30
MSDL [15]	83.20	80.65	79.50	68.44
D2L2R2 [4]	92.20	90.40	91.30	75.30
SLRDL [5]	87.35	83.40	82.47	72.30
JP-LRDL [3]	93.20	93.00	93.30	78.23
AlexNet [16]	30.33	30.12	30.17	25.55
VGG-Face [17]	85.90	85.01	87.30	79.83
MM-SLDL	96.70	96.41	96.30	85.30

Table 2. Table 2 : Recognition rates (%) on LFWa dataset

Method	Rec. Rate	Method	Rec. Rate
MLDL [9]	74.10	UMD2L [8]	70.43
MSDL [15]	64.25	D2L2R2 [4]	75.20
SLRDL [5]	74.20	JP-LRDL [3]	79.87
AlexNet [16]	40.31	VGG-Face [17]	90.01
SLDL-Mod2	76.77	MM-SLDL	88.04

Equations

$$\underset{D_K,Z_K,E_K}{\min }\sum _{K=1}^2(\Vert Z_K{\Vert }_{\ast }+\beta \Vert Z_K{\Vert }_1+\lambda \Vert E_K{\Vert }_1)$$

$$+\alpha \Vert Z_1{Z}_2^T-Q{\Vert }_F^{2}s.t.X_K=D_K{Z}_K+E_{K}K=1,2$$

$$\widehat{W_K}=\underset{W_K}{\mathrm{argmin}}\Vert H-W_K{Z}_K{\Vert }_2^2+\lambda \Vert W_K{\Vert }_2^2$$

$$\displaystyle c_{K}=\operatorname*{argmax\>}_{c_{K}}\big{(}s=(\hat{W_{K}}+Q)Z_{K,i}^{ts}\big{)}$$

$$\displaystyle\lVert L(c_{K})-\big{(}X_{K,i}^{ts}-\bar{S}(c_{K})\big{)}\rVert_{F}^{2}$$

$$\underset{D_K,Z_K,E_K}{\min }\sum _{K=1}^2(\Vert Z_K{\Vert }_{\ast }+\beta \Vert W_K{\Vert }_1+\lambda \Vert E_K{\Vert }_1)$$

$$+\alpha \Vert Z_1{Z}_2^T-Q{\Vert }_F^{2}s.t.X_K=D_K{Z}_K+E_K,W_K=Z_K$$

$$L=\Vert Z_K{\Vert }_{\ast }+\beta \Vert W_K{\Vert }_1+\lambda \Vert E_K{\Vert }_1+\alpha \Vert Z_1{Z}_2^T-Q{\Vert }_F^2$$

$$+<Y_K,X_K-D_K{Z}_K-E_K>+<M_K,Z_K-W_K>$$

$$+\frac{\mu }{2}(\Vert X_K-D_K{Z}_K-E_K{\Vert }_F^2+\Vert Z_K-W_K{\Vert }_F^2)K=1,2$$

$$\Vert Z_K{\Vert }_{\ast }+\beta \Vert W_K{\Vert }_1+\lambda \Vert E_K{\Vert }_1+\alpha \Vert Z_1{Z}_2^T-Q{\Vert }_F^2$$

$$+\frac{\mu }{2}(\Vert X_K-D_K{Z}_K-E_K+\frac{Y_K}{\mu }{\Vert }_F^2$$

$$+\Vert Z_K-W_K+\frac{M_K}{\mu }{\Vert }_F^2)-\frac{1}{2\mu }(\Vert Y_K{\Vert }_F^2+\Vert M_K{\Vert }_F^2)$$

$$Z_K^{j+1}=\underset{Z_K}{\mathrm{argmin}}\frac{1}{\eta \mu }\Vert Z_K{\Vert }_{\ast }+\frac{1}{2}\Vert Z_K-Z_K^j+\mu [-D_K^T(X_K$$

$$-D_K{Z}_K^j-E_K^j+\frac{Y_K^j}{\mu })+\frac{2\alpha (Z_K{Z}_l^T-Q)Z_l}{\eta \mu }$$

$$+\mu (Z_K-W_K^j+\frac{M_K^j}{\mu })]/\eta \mu {\Vert }_F^2\text{where}l\ne K$$

$$W_K^{j+1}=\underset{W_K}{\mathrm{argmin}}\frac{\beta }{\mu }\Vert W_K{\Vert }_1+\frac{1}{2}\Vert W_K-Z_K^{j+1}-\frac{M_K^j}{\mu }{\Vert }_F^2$$

$$E_K^{j+1}=\underset{E_K}{\mathrm{argmin}}\frac{\lambda }{\mu }\Vert E_K{\Vert }_1+\frac{1}{2}\Vert E_K-(\frac{Y_K^j}{\mu }+X_K$$

$$-D_K{Z}_K^{j+1}){\Vert }_F^2$$

$$\frac{\mu }{2}(\Vert X_K-D_K{Z}_K-E_K+\frac{Y_K}{\mu }{\Vert }_F^2+\Vert Z_K-W_K{\Vert }_F^2)$$

$$+C(Z_K,W_K,E_K,Q)$$

$$D_K^{j+1}=\gamma D_K^j+(1-\gamma )D_K^{update}$$

Full text

**Index Terms— ** Multi-modal dictionary learning, Low-rank learning, Illumination invariant, Face recognition

1 INTRODUCTION

The last decade has witnessed a tremendous progress in face recognition technologies, and great recognition performance has been reported by different methods under some ideal conditions, but most of these methods are not robust to outliers, occlusions, severe illumination and pose variations. In recent years, dictionary learning (DL) algorithms have been successfully applied to different vision tasks including face recognition. DL is a feature learning technique in which, an input signal is represented with a sparse linear combination of dictionary atoms. To alleviate the effects of aforementioned variations, low-rank (LR) matrix recovery has been integrated into the DL framework, and is shown to achieve promising results when corruption existed. LR matrix recovery [1] was oroginally proposed to recover a LR matrix from corrupted observations, and have succesfuly been applied to applications like background modeling [2] and image classification [3]. Li et al. [4] developed a discriminative DL method by combination of the Fisher discrimination and the LR constraint on sub-dictionaries. Zhang et al. [5] presented a structured, sparse and LR representation for image classification by adding a regularization term to the DL objective function. Recently, Foroughi et al. [3] proposed a joint projection and LR-DL method using dual graph constraints for classification of small datasets, which include considerable amount of variations.

In parallel developments, it is well established that information fusion using multiple sources can generally improve the recognition performance, since it provides a framework to combine information from different perspectives that is more tolerant to the errors of individual sources [6]. To benefit from information fusion, some methods have also successfully incorporated DL technique into the feature learning framework. Monaci et al. [7] proposed a multi-modal DL algorithm to extract typical templates, which represents synchronous transient structures between multi-modal features. [8] proposed an uncorrelated multi-view discrimination DL method based on the Fisher discrimination, that jointly learns multiple uncorrelated discriminative dictionaries from different views. Nevertheless, the only work that integrated LR into multi-modal DL was presented by Wu et al. [9] through constructing class-specific sub-dictionaries for each modality, and utilizing LR and incoherence constraints on each view.

To construct different modalities, most of the existing methods either exploit multi-view angles [9] or extract different local features [9] or weak biometrics [10] from pre-defined regions of face images. These methods suffer from two main disadvantages that burden extra overhead on the system. Firstly, they demand either several cameras or manual region definition and hand-crafted feature extraction and secondly, they are not applicable to millions of available face data that have already been captured under single view. By exploiting more meaningful modalities, we address these challenges and even increase the recognition rate further. Recently, Shakeri et al. [11] presented an illumination invariant representation of an image for outdoor place recognition. To create this representation, they use a Wiener filter derived from the power law spectrum assumption of natural images that is robust against illumination variations. Since the obtained representation may lose the chromaticity of the image, a shadow removal method based on entropy minimization is utilized. This representation showed superior performance for outdoor place recognition in various illumination and shadow variations. Inspired by this success, we design a framework for multi-modal fusion with the following contributions:

• •

We design a multi-modal LR-DL method, where in each modality a discriminative and reconstructive dictionary, and a structured sparse and LR representation are learned from face images, and the collaboration between modalities is encouraged by incorporating an ideal representation term. We provide a new classification schema, which utilizes the reconstruction by LR and sparse noise components.

• •

By adopting illumination invariant representation of images as one of the modalities, the model learns robust and discriminative representations from noisy images, even when the kind of variation is different in the training and test sets. The proposed method achieves superior performance for small datasets that have large intra-class variation.

2 THE PROPOSED MM-SLDL METHOD

We propose a Multi-Modal Strcutured Low-rank Dictionary Learning method (MM-SLDL) for face recognition, in which we use two modalities. While the first modality is constructed by the raw pixels of face images, the second is formed by illumination invariant images [11]. Denote $X_{K}\>(K=1,2)$ the training data from the $K^{th}$ modality including $C$ classe, as $X_{K}=\{X_{K}^{1},X_{K}^{2},\dots,X_{K}^{C}\}$, where $X_{K}^{i}$ corresponds to class $i$ in the $K^{th}$ modality. In each modality, we use a supervised learning method to learn a discriminative and reconstructive dictionary $D_{K}$, and a structural sparse and LR image representation $Z_{K}$. LR matrix recovery helps to decompose the corrupted matrix $X_{K}$ into a LR component $DZ$ and a sparse noise component $E$, i.e., $X_{K}=D_{K}\,Z_{K}+E_{K}$. With respect to dictionary $D_{K}$, the optimal representation matrix $Z_{K}$ for $X_{K}$ should be block-diagonal [12], i.e., $Z^{\ast\,ii}_{K}=Z_{K}^{i}$. In each modality, the dictionary $D_{K}$ contain $C$ sub-dictionaries as $D_{K}=\{D_{K}^{1},D_{K}^{2},\dots,D_{K}^{C}\}$, where $D_{K}^{i}$ corresponds to the $i^{th}$ class. Let $Z_{K}^{i}=\{Z_{K}^{i,1},Z_{K}^{i,2},\dots,Z_{K}^{i,C}\}$ be the representation of $X_{K}^{i}$ with respect to $D_{K}$, then $Z_{K}^{i,j}$ denotes coefficients for $D_{K}^{j}$. To learn robust representations from images, $D_{K}$ should have discriminative and reconstructive power. Firstly, $D_{K}^{i}$ should well represent the samples in class $i$, and ideally be exclusive to each subject $i$. Secondly, every class $i$ needs to be well represented by its sub-dictionary, such that $X_{K}^{i}=D_{K}^{i}\,Z_{K}^{i,i}+E_{K}^{i}$, and finally $Z_{K}^{i,j}$, the coefficients for $D_{K}^{j}\,(i\neq j)$, are nearly all zero. So, the objective function of MM-SLDL is defined as:

[TABLE]

The main objective function simultaneously trains two dictionaries and representations under the joint ideal regularization prior. $Q$ is an ideal representation built from training data in block-diagonal form, and defined as $Q=\{q_{1},q_{2},\dots,q_{\mathcal{C}}\}\in R^{\mathcal{C}\times\mathcal{C}}$. Here $\mathcal{C}$ is the size of dictionary, and $q_{i}$ is the code for sample $x_{K}^{i}$ in the form of $[0\dots p_{i},p_{i},p_{i},\dots]^{t}\in R^{\mathcal{C}}$, where $p_{i}$ is the number of training samples in class $i$. This means that if $x_{K}^{i}$ belongs to class L, then the coefficients in $q_{i}$ for $D_{K}^{\textit{L}}$ are all $p_{i}$s, while the others are all [math]s. We add the regularization term $\lVert Z_{1}Z_{2}^{T}-Q\rVert_{F}^{2}$ for two reasons: first, to include the structure information into the dictionary learning process and second, to enforce collaboration between two modalities. It encourages the training images of the same class to have the same representation $Z_{K}$ in different modalities, despite of intra-class variations.

Classification Scheme: After dictionaries $D_{K}$ are learned, the LR sparse representations $Z_{K}$ of training data $X_{K}$ and $Z_{K}^{ts}$ of test data $X^{ts}$ are calculated by solving (2) separately using Algorithm 1 with $\alpha=0$. The representation $Z_{K,i}^{ts}$ of the $i^{th}$ test sample is the $i^{th}$ column of $Z_{K}^{ts}$. Using the multivariate ridge regression model [13], we obtain a linear classifier $\hat{W_{K}}$ as:

[TABLE]

where $H$ is the class label matrix of $X_{K}$. This yields $\hat{W}=HZ_{K}^{T}(Z_{K}Z_{K}^{T}+\lambda I)^{-1}$. The estimated label of the $K^{th}$ modality is obtained as:

[TABLE]

where $s$ is the class label vector. We then use LR matrix recovery to obtain LR and sparse noise components of potential classes $c_{1},c_{2}$, and then compute the reconstruction error of the given query sample $X_{i}^{ts}$ in both modalities:

[TABLE]

where $L(c_{K})$ is the LR component of class $c_{K}$ in the $K^{th}$ modality, and $\bar{S}(c_{K})$ is the average sparse noise of that class. Since the data range is different in two modalities, we use a normalization step, and the winner class is the one that minimizes the ratio of (4) between two modalities.

3 OPTIMIZATION OF MM-SLDL

In each iteration we update the variables of the $K^{th}$ modality, while fixing the variables of other modality, and for the $K^{th}$ modality, the variables are updated alternatively. To solve optimization problem (2), we first introduce an auxiliary variable $W_{K}$ to make it separable:

[TABLE]

The augmented Lagrangian function $L$ of (5) is defined as:

[TABLE]

[TABLE]

where $<A,B>=\textit{tr}(A^{T}B)$, $Y_{K}$ and $M_{K}$ are Lagrange multipliers and $\mu$ is a balance parameter. The optimization problem (3) can be divided into two sub-problems as follows:

• •

Updating Coding Coefficient $Z_{K}$: With $D_{K}$ fixed, we use the linearized alternating direction method with adaptive penalty (LADMAP) [14] to solve for $Z_{K}$ and $E_{K}$. The augmented Lagrangian function (3) would reduce to:

[TABLE]

The function (• ‣ 3) should be minimized by alternative updating variables $Z_{K},W_{K},E_{K}$ as follows:

[TABLE]

where $\eta=\lVert D_{K}\rVert_{2}^{2}$, and we notice that $Q=Q^{T}$.

[TABLE]

[TABLE]

• •

Updating Dictionary $D_{K}$: When $Z_{K},W_{K},E_{K}$ are fixed, we would be able to update $D_{K}$. The Lagrangian function (3) is further reduced to:

[TABLE]

where $C(Z_{K},W_{K},E_{K},Q)$ is fixed. Equation (• ‣ 3) is in the quadratic form and $D_{K}$ can be solved directly as follows:

[TABLE]

where $D_{K}^{update}=\frac{1}{\mu}(Y_{K}+\mu(X_{K}-E_{K}))Z_{K}^{T}(Z_{K}Z_{K}^{T})^{-1}$. We initialize the dictionary using KSVD method on training samples of each class and combining all the classes.

4 RESULTS AND DISCUSSION

The performance of MM-SLDL method is evaluated on three face datasets. We compare our method with three types of methods: (1)Multi-modal LR-DL method MLDL [9] (2)Multi-modal DL methods including UMD2L [8] and MSDL [15] (3)Single-modality LR-DL methods such as JP-LRDL [3], D2L2R2 [4] and SLRDL [5]. For constructing the training set, we select images randomly and the selection is repeated $10$ times and we report the average recognition rates for all methods. We set the number of dictionary atoms of each class as training size, and choose the tuning parameters of all methods by 5-fold cross validation.

Convolutional Neural Networks (CNNs) have significantly improved the face recognition rates, and the most important ingredient for the success of such methods is the availability of large quantities of training data; however, transfer learning is a powerful tool to train small target datasets. [18] revealed when the target dataset is small and similar to original dataset, it is better to treat CNN as fixed feature extractor and train a linear classifier on the CNN features. We compare MM-SLDL with two deep methods (1)Deep features generated by VGG-Face descriptor [17], that is based on a $16$-layer CNN trained on $2.6$M images, followded by a nearest neighbor classifier. (2)We use $8$-layer AlexNet [16] trained on $1.2$M images of the ImageNet dataset, and fine-tune it on the target data.

AR Dataset [19] includes over $4,000$ face images from $126$ individuals, $26$ images for each person in two sessions. Among the images of each session, $3$ are obscured by scarves, $3$ by sunglasses, and the remaining faces are of different facial expressions or illumination variations, which we refer to as unobscured images. Following [5], experiments are conducted under three scenarios. Sunglasses: We select $7$ unobscured images and $1$ with sunglasses from the first session as training samples for each person, and the rest of unobscured and sunglasses images are used for testing. Scarf: We choose $8$ training images ($7$ unobscured and $1$ with scarf) from the first session for training, and $12$ test images including the rest of unobscured and scarf images. Mixed: We select $7$ unobscured, plus $2$ occluded images ($1$ with sunglasses, $1$ by scarf) from the first session for training, and the remaining $17$ images in two sessions for testing. We design a challenging scenario Misc., in which we select $7$ unobscured, and $1$ scarf images from the first session for training, and utilize the remaining $7$ unobscured and $6$ sunglasses images for testing. Here, the type of noise is different in training and test sets. According to Table 1, MM-SLDL achieves the best performance in all scenarios, and the improvement is significant in “Misc.” scenario, where all the other methods fail.

Fig. 1b illustrates examples of image decomposition on AR dataset. The first and second rows show training images and learned LR component $D_{K}Z_{K}$ in two modalities. While the first modality keeps more details, the illumination invariant modality better separates occlusions from the original images; hence, a robust representation is learned by their fusion. Fig. 1a demonstrates a testing sample $x_{ts}$, and $x_{ts}-\bar{S}(c_{K}),L(c_{K})$ components, which their difference determines the winner, that is illustrated by a red tick mark.

Extended YaleB Dataset [20] contains $2,414$ face images of $38$ human subjects captured under different illumination conditions. There are $59\sim 64$ images for each subject, and we randomly select $20$ of them for training. We simulate various levels of contiguous occlusion from $20\%$ to $60\%$, by replacing a randomly located square block of each train image with an unrelated image, as seen in Fig. 3a. To have a real challenge, test images are not occluded. We visualize the representation $Z$ for two modalities for testing images of the first $10$ classes under $40\%$ occlusion training scenario in Fig. 2a. Testing images automatically generate a block diagonal structure, and the second modality learns a better representation here. Fig. 2b illustrates the recognition rates of all methods across different occlusion levels, and MM-SLDL outperforms other counterparts, especially for severely occluded images.

LFW Dataset [21] contains $13,233$ unconstrained face images of $5,749$ different individuals, collected from the web with large variations in pose, expression, illumination, clothing, hairstyles, occlusion, etc. We use an aligned version of LFW called LWFa [22], and exploit $143$ subject with no less than $11$ samples per subject to perform the experiment. Some of these images are shown in Fig. 3b. A central $170\times 140$ region is cropped from each of images and the first $10$ samples per class are selected for training, while the rest is used for testing. Table 2 shows the recognition rates of all compared methods. Although VGG-Face has already been trained on extra $2.6$M images, MM-SLDL achieves competitive results using too much smaller training data, which have large intra-class variation. Also, as expected fine-tuned AlexNet is prone to overfitting because target data is small and very different in content compared to the ImageNet. Finally, to verify the role of illumination invariant modality, we just use $K=2$ in the objective function (2) and change the ideal representation term to $\lVert Z_{K}-Q\rVert_{F}^{2}$. The results are reported under SLDL-Mod2, and as observed is not competitive.

5 CONCLUSIONS

We proposed a face recognition method that learns discriminative dictionaries and structured sparse LR representations from contaminated face image in two modalities. Adopting the illumination invariant representation of images as a modality, additionally empowers the model. Experimental results indicate that MM-SLDL is robust, achieving state-of-art performance in the presence of occlusion, illumination and pose changes, using a few training samples.