A Boost in Revealing Subtle Facial Expressions: A Consolidated Eulerian Framework

TL;DR

This paper introduces a consolidated Eulerian framework that jointly amplifies and expands micro-expression signals, improving the accuracy and efficiency of facial micro-expression recognition over existing methods.

Contribution

It proposes a unified approach that combines motion magnification and time interpolation into a single framework, capturing their underlying connections for better performance.

Findings

Outperforms state-of-the-art in accuracy on public MER databases.

Achieves faster processing of micro-expression clips.

Enhances distinguishability of subtle facial movements.

Abstract

Facial Micro-expression Recognition (MER) distinguishes the underlying emotional states of spontaneous subtle facialexpressions. Automatic MER is challenging because that 1) the intensity of subtle facial muscle movement is extremely lowand 2) the duration of ME is transient.Recent works adopt motion magnification or time interpolation to resolve these issues. Nevertheless, existing works dividethem into two separate modules due to their non-linearity. Though such operation eases the difficulty in implementation, itignores their underlying connections and thus results in inevitable losses in both accuracy and speed. Instead, in this paper, weexplore their underlying joint formulations and propose a consolidated Eulerian framework to reveal the subtle facial movements.It expands the temporal duration and amplifies the muscle movements in micro-expressions simultaneously. Compared toexisting approaches, the proposed method can not only process ME clips more efficiently but also make subtle ME movementsmore distinguishable. Experiments on two public MER databases indicate that our model outperforms the state-of-the-art inboth speed and accuracy.

Figures

Tables

Table 1. Table 1: Experimental results of MER comparing to recent state-of-the-art approaches on the SMIC and the CASMEII databases.

	SMIC-HS	CASMEII
Baseline-SMIC [2]	48.8%	-
Baseline-CASMEII [3]	-	63.41%
Adaptive Mag [26]	-	51.9%
Wang et al.[39]	-	62.3%
Liong et al.[40]	53.56%	-
Huang et al.[34]	60.98%	64.37%
DiSTLBP-RIP [33]	63.41%	64.78%
Patel et al.[13]	53.60%	47.30%
Li et al.[7]	68.29%	67.21%
Ours	68.90%	70.85%

Table 2. Table 2: Experimental results of MER on sub-datasets of SMIC.

	SMIC-subHS	SMIC-VIS	SMIC-NIR
Baseline-SMIC [2]	-	52.1%	38.0%
Li et al.[7]	80.28%	81.69%	67.61%
Ours	87.32%	83.10%	69.01%

Equations

$$\hat{V}=f(V;\Theta ),$$

$$\hat{V}=V\mathbf{W}.$$

$$\mathbf{W}={\mathbf{W}}^M{\mathbf{W}}^I,$$

$$I_m(t)=\sum _{i=1}^{T}I(i)W_{i,t}^M.$$

$$s(x+\delta (t))=s(x)+\delta (t)\frac{\partial s}{\partial x},$$

$$\mathbf{B}(t)=L_1(t)-L_2(t).$$

$$\{\begin{array}{l}{L}_1(t)=w_{1}I(t)+(1-w_1)L_1(t-1)\\ L_2(t)=w_{2}I(t)+(1-w_2)L_2(t-1),\end{array}$$

$$I_m(t)=I(t)+\alpha \mathbf{B}(t).$$

$$W_{i,j}^M=\{\begin{array}{ll}\alpha (1-w_1{)}^a{w}_1^b-\alpha (1-w_2{)}^a{w}_2^b& \text{if j>i }\\ \alpha (w_1-w_2)+1& \text{if j=i}\\ 0& \text{else},\end{array}$$

$$\hat{V}=V_m{\mathbf{W}}^I.$$

$${\mathcal{F}}_{map}(I_m(t))=({\mathbf{M}}^{-1}{\mathcal{V}}^{\top }{\mathbf{U}}^{\top })(I_m(t)-\overline{I_m}),$$

$$V_m=\mathbf{AY}+\overline{V_m},$$

$$\hat{V}=\mathbf{A}{\mathbf{Y}}^{\prime }+\overline{V_m^{\prime }},$$

$$\mathbf{A}=V_m(\mathbf{I}-\frac{1}{T}{1}_{T\times T}){\mathbf{Y}}^{\top }(\mathbf{Y}{\mathbf{Y}}^{\top }{)}^{-1}.$$

$$\hat{V}=V_m(\mathbf{I}-\frac{1}{T}{1}_{T\times T}){\mathbf{Y}}^{\top }(\mathbf{Y}{\mathbf{Y}}^{\top }{)}^{-1}{\mathbf{Y}}^{{}^{\prime }}+\overline{V_m^{\prime }}.$$

$${\mathbf{W}}^I=(\mathbf{I}-\frac{1}{T}{1}_{T\times T}){\mathbf{Y}}^{\top }(\mathbf{Y}{\mathbf{Y}}^{\top }{)}^{-1}{\mathbf{Y}}^{{}^{\prime }}+\frac{1}{T}{1}_{T\times T^{\prime }}.$$

Full text

Recent works adopt motion magnification or time interpolation to resolve these issues. Nevertheless, existing works divide them into two separate modules due to their non-linearity. Though such operation eases the difficulty in implementation, it ignores their underlying connections and thus results in inevitable losses in both accuracy and speed. Instead, in this paper, we explore their underlying joint formulations and propose a consolidated Eulerian framework to reveal the subtle facial movements. It expands the temporal duration and amplifies the muscle movements in micro-expressions simultaneously. Compared to existing approaches, the proposed method can not only process ME clips more efficiently but also make subtle ME movements more distinguishable. Experiments on two public MER databases indicate that our model outperforms the state-of-the-art in both speed and accuracy.

Introduction

††footnotetext: W.Peng, X. Hong, Y. Xu, and G. Zhao are with Center for Machine Vision and Signal Analysis, University of Oulu, Finland This work has been accepted by 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition FG 2019. Copyright @ 2019 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE.

Micro-expression recognition (MER) is an emerging facial affective analysis task to distinguish the subtle spontaneous facial expressions. Automatic MER has many potential applications like clinical diagnosis, lie detection, and human computer interaction (HCI). MER is challenging since people tend to hide their emotions in case of being found. Studies of ME in psychology indicate that even a trained human can only achieve a MER accuracy lower than $50\%$ [1].

Promising progresses have been made over the past decade, in aspects such as dataset construction [2, 3, 4], spatial-temporal descriptors either by hand-crafting [5, 6, 7, 8, 9, 10, 11, 12], or deep learning [13, 14, 15, 16, 17], [18, 19], etc. However, it is still far away from satisfaction. The difficulties are two-fold : 1). The ME related muscle movements are remarkably subtle. ME, unlike general facial expressions, only covers a small area of face so that the movements are too imperceptible to be detected. 2). ME duration is quite short. Generally, the temporal duration of ME only lasts approximately from 1/25 to 1/3 seconds [20].

Recent works tend to employ motion magnification or time interpolation as pre-processing steps for MER. Motion MAGnification (MAG) [21, 22, 23] amplifies the intensity of motion and makes subtle motion much easier to observe. Li et al. [7] amplified the subtle facial movements by employing MAG before extracting features and improved the accuracy by around $10\%$ on the CASMEII [3] database. Obvious improvements are also observed in [24, 25, 26] as a result of introducing MAG. Besides MAG, time interpolation also attracts increasing attention. Pfister et al. [27] proposed to use Time Interpolation Model (TIM) [28, 29] for MER problem, which allows to obtain sufficient frames for feature extraction even for very short expressions. Methods [2, 3, 30, 31, 7, 32, 33, 34] also followed a similar mechanism and got promising results.

Despite these progresses, MAG and TIM are usually treated as two separate modules [3, 34, 7]. Such assumption eases the difficulty in implementation. However, there are several problems. Firstly, it is prone to having the diluted problem††A phenomenon that the facial muscle movements fall into decline. [30] into the succeeding process. Secondly, the calculation of intermediate products increases the computational costs. Finally, what makes it worse, ignoring the underlying connections will inevitably result in the loss of recognition accuracy.

To address these issues, we explore the underlying joint formulations for MAG and TIM and propose a consolidated framework for revealing the subtle spatial-temporal information in ME clips.The proposed model, which is called ME-Booster, can accomplish temporal interpolating and facial muscle motion magnification simultaneously. Compared to the traditional MER systems, our ME-Booster not only avoids unnecessary separation of the two modules but also eliminates the side effect brought by the intermediate process. We investigate our framework on two popular spontaneous MER databases: SMIC [2] and CASMEII [3]. Experimental results indicate that our model is computationally economic and substantially outperforms the state-of-the-art approaches.

The contributions of this paper are two-fold: 1. We propose a consolidated video motion revealing model for MER, where TIM and MAG are, for the first time, jointly formulated. More importantly, ME-Booster is a distillation of the two modules, beyond a simple knockoff just putting them together; 2. We provide fast implementations of MAG and TIM methods by exploring their underlying linear forms. It is thus computationally efficient.

1 Methodology

In this section, we describe a consolidated Eulerian subtle motion revealing model, termed ME-Booster, to boost micro-expression recognition. Suppose there is a ME video clip $V=\left(I(1),I(2),\dots,I(T)\right)$, where $I(t)\in\mathbb{R}^{d}$ denotes the $t$-th frame of the video for $t=1,\cdots,T$. Our goal is to produce a video $\hat{V}$ where the micro-expression can be easily observed.

The model, in its general form, is formulated by a video-specific function $f$ with a set of parameters $\Theta$ as follows:

[TABLE]

where the function $f:\mathbb{R}^{d\times T}\mapsto\mathbb{R}^{d\times{T}^{\prime}}$ maps the input video to a fixed length ($T^{\prime}$) one of which the subtle facial muscle movements in $V$ are revealed.

Considering the non-linear nature of the representative motion magnification and temporal interpolation methods, such as the Eulerian based MAG method [22] and the graph spectral TIM method [35], existing MER approaches [3, 34, 7] usually treat MAG and TIM as two separate modules and adopt them in a successive manner to alleviate the difficulty in implementation. As discussed above, it may easily lead to losses in both accuracy and speed.

To the contrary, in this paper we will show that under certain but reasonable circumstances, MAG and TIM can be embedded in Eq. (1) by a scale-wise linear function. Firstly, the Eulerian MAG model [21, 22, 23] is multi-scale and non-linear, and thus a complicated system. However, we realize that it is feasible to implement it as a series of scale-wise linear functions, since in each of the scales it approximates movements using the Taylor expansion. Secondly, we will show that the interpolated video can be obtained by multiplying the input video with a matrix, without explicitly solving the generalized eignvalue problem as the graph spectral TIM method [28] does. Thus, w.l.o.g., in the following context, we formulate our model with a ‘single-resolution’ setting and consider the $f(\cdot)$ as a linear function:††Multi-resolution analysis can be easily reached by applying the model to each single resolution.

[TABLE]

To obtain the parameter matrix $\mathbf{W}\in\mathbb{R}^{T\times T^{{}^{\prime}}}$, we decouple it as follow:

[TABLE]

in which $\mathbf{W}^{M}\in\mathbb{R}^{T\times T}$ and $\mathbf{W}^{I}\in\mathbb{R}^{T\times T^{{}^{\prime}}}$ are the magnification matrix and the interpolation matrix, respectively. The order of the right hand side of Eq. (3) is consistent with the one in [7].

In the following paragraphs, we will derive the solutions of $\mathbf{W}^{M}$ and $\mathbf{W}^{I}$ respectively in detail and show that both $\mathbf{W}^{M}$ and $\mathbf{W}^{I}$ can be determined once the lengths of the input and output video are given. As a result, Eqs. 2 and 3 ensure that $\mathbf{W}$ is obtained before being applied on the input video in practice.

Solution to the magnification matrix $\mathbf{W}^{M}$

In this subsection, we aim to get the solution for $\mathbf{W}^{M}$. Let $W^{M}_{i,j}$ be the entry of $\mathbf{W}^{M}$ at $\left(i,j\right)$ for $i,j=1,\cdots,T$. The magnified frame $I_{m}(t)$ can be described as :

[TABLE]

To get the solution of $\mathbf{W}^{M}$, we revisit the Eulerian magnification model [22]. It regards the first frame in a video as a digital signal $s(x)$, and every frame as a displacement of this signal, i.e., $I(x,t)=s(x+\delta(t))$, where $I(x,t)$ is the pixel intensity at location $x$ of the $t$-th frame and $\delta(t)$ is a tiny displacement function. It is then approximated by the first-order Taylor expansion :

[TABLE]

where every frame in the ME video is an addition of the base signal and the multiplication of the displacement and the first derivative. Here, $\frac{\partial s}{\partial x}=I(t)-I(t-1)$ represents the ME motion. It is fulfilled using a recursive algorithm to take all the preceding frames into account and get the ‘historical’ motion $\mathbf{B}(t)$:

[TABLE]

$L_{1}(t)$ and $L_{2}(t)$, for $t=2,\cdots,T$ are auxiliary variables:

[TABLE]

where the $L_{1}(1)=I(1),\leavevmode\nobreak\ L_{2}(1)=I(1)$. The hyper-parameters $w_{1}$ and $w_{2}$ are in the range $(0,1)$ and $w_{1}>w_{2}$. Then applying a magnification factor $\alpha$ to $\mathbf{B}(t)$, we get the magnified frame $I_{m}(t)$ :

[TABLE]

The Eulearian magnification model in Eq. (8) is clearly a linear model. We solve out a parameter matrix operated on the input video by expanding the recursive form in (7). As a result, by combining Eq. (4) and Eq. (8), we get the solution for $\mathbf{W}^{M}$:

[TABLE]

where $a={j-i}$ and $b={min(1,i-1)}$. As it is in Eq. (9), $\mathbf{W}^{M}$ is an upper triangular matrix since ME motion is only related to the preceding video frames. Once the lengths of the input and output videos are determined, the entry of $\mathbf{W}^{M}$ can be computed and stored by a look-up table (LUT). That will further improve the computational efficiency. To get a better magnification result, one can employ $\mathbf{W}^{M}$ on the multi-scale input video, where for different scales, the factor $\alpha$ in Eq. (9) may be truncated.

**Solution to the interpolation matrix $\mathbf{W}^{I}$ **

In this subsection, we will derive $\mathbf{W}^{I}$ in Eq. (3) to form our efficient time interpolation module for extending $V_{m}$ to $T^{\prime}$ frames. Just for clarification, we introduce the symbols $V_{m}=\{I_{m}(t)\}$ for $t=1,2,\cdots,T$ to describe the intermediate magnification output, though they are finally eliminated by Eq. 3 in practice. The recovered video $\hat{V}$ is:

[TABLE]

Applying a graph spectral method [35], the traditional TIM approach [28] finds projection functions by which TIM keeps the frame adjacency priority in the low latent space. That is

[TABLE]

where the function $\mathcal{F}_{map}:\mathbb{R}^{d}\mapsto\mathbb{R}^{T-1}$ describes a projection from an image to $T-1$ points. Here, $\mathbf{U}$ is an unitary matrix from SVD [36] on input, $\mathbf{M}$ and $\mathcal{V}$ are graph Laplacian related diagonal matrix and eigenvectors, respectively. $\bar{I_{m}}$ is the average frame of all the frames in video $V_{m}$. Readers may refer to [28] for more details.

In this way, every frame becomes a group of discrete points in a latent space. According to [35], the optimal projection is a group of sine functions. Let $\mathcal{F}(t)=\mathcal{F}_{map}(I_{m}(t))$. By sampling from the latent space and projecting the points back to image space, we reconstruct the input. According to Eq. (11), the intermediate magnification video can be reconstructed by

[TABLE]

where $\bar{V_{m}}=V_{m}(\frac{1}{T}\mathbf{1}_{T\times T})$, $\mathcal{V}^{-1}$ exists since $\mathcal{V}$ is full rank. So $\mathbf{A}=\mathbf{U}(\mathcal{V}^{-1})^{\top}\mathbf{M}$, and $\mathbf{Y}=[\mathcal{F}(\frac{1}{T}),\mathcal{F}(\frac{2}{T}),...,\mathcal{F}(1)]$ are $T$ groups of points in latent space. Each $\mathcal{F}(t)$, which are sampled from $T-1$ sine functions, refers to one frame. So sampling $T^{{}^{\prime}}$ groups of points, say $\mathbf{Y}^{{}^{\prime}}=[\mathcal{F}(\frac{1}{T^{{}^{\prime}}}),\mathcal{F}(\frac{2}{T^{{}^{\prime}}}),...,\mathcal{F}(1)]$ , the video is extended to $T^{{}^{\prime}}$ frames. That is :

[TABLE]

where $\bar{V_{m}^{\prime}}(t)=V_{m}(\frac{1}{T}\mathbf{1}_{T\times T^{\prime}})$. Since $\mathbf{Y^{\prime}}$ is a group of discrete points, it satisfies the requirement for a linear TIM model. According to Eq. (12), we adopt the right inverse matrix to compute $\mathbf{A}$ :

[TABLE]

Substituting Eq. (14) into Eq. (13), we have

[TABLE]

Finally we obtain the solution for $\mathbf{W}^{I}$, that is:

[TABLE]

Here Eq. (16) provides us a linear solution to $\mathbf{W}^{I}$ and it is independent to the input video, which means $\mathbf{W}^{I}$ also can be computed in advance.

After obtaining $\mathbf{W}^{M}$ and $\mathbf{W}^{I}$, we employ Eq. (3) to solve the matrix $\mathbf{W}$, which is the only parameter matrix used in practice. Finally, a consolidated model is constructed by using this linear operator $\mathbf{W}$. Obviously, ME-booster is a unique model for both video magnification and time interpolation. The relation to the magnification and interpolation method is clear. When $\mathbf{W}^{I}$ is an identity matrix, the model formulated in Eq. (2) is a MAG model; When $\mathbf{W}^{M}$ is an identity matrix, it degenerates to a TIM model.

Our ME-booster model is computationally economic and can make the subtle ME more distinguished. These advantages are mainly contributed by the discovery of the underlying common linear property between TIM and MAG modules. By this way, there are no need of intermediate process and complicated computation of graph Laplacian. Besides, expanding recursive forms and decoupling model parameters from the input also dedicate to these advantages.

2 Experiments

We evaluate the proposed model on two widely used spontaneous ME databases: SMIC [2] and CASMEII [3]. The SMIC database contains 306 ME video clips belonging to four sub-sets, which are SMIC-HS, SMIC-VIS, SMIC-NIR, and SMIC-subHS. There are three classes for all the ME clips : Positive, Negative, and Surprise. The CASMEII database contains 247 ME video clips. The baseline method of CASMEII chooses classes *Happiness, Surprise, Disgust, Repression, *and Others for training and testing.

To extensively evaluate the proposed ME-Booster, we perform the following three experiments: A). Consolidation versus separate; B). Comparisons to the state of the arts; and C). Computational cost evaluation.We explore the different values for hyper-parameters $T^{\prime}$, $\alpha$, and $w_{1}$ and $w_{2}$. Based on the results, which are found consistent to those in [7, 22], the parameters are empirically set to 10, 16, 0.4, and 0.05, respectively. For the experiments A) and B), a linear SVM [37] is used as classifier for our model, unless further specified. As suggested by [7], in all MER experiments, the Leave-One-Subject-Out (LOSO) protocol is adopted. The micro-average method [38] is employed as evaluation metrics.

A. Consolidation versus separate

We compare ME-Booster with the current best method [7], which contains both the MAG and TIM modules as two separate ones, as illustrated as the ’traditional model’ in Fig. 1. For fair comparison, we use the same classifier and three mainstream temporal-spatial feature descriptors: LBP-TOP[6], HOG-TOP[5] and HIGO-TOP [7], which are extensively evaluated in the state of the art [7].

The comparative results are reported in Fig. 2. There are several important observations. 1) For HIGO-TOP, the accuracy is improved by almost 10.0% on the SMIC-NIR. On the SMIC-subHS, our accuracy is 87.32% , while the current best is only 80.28%. 2) For the HOG-TOP, our model improves the accuracy by 3.0% on average. 3) For the LBP-TOP, ME-Booster is slightly better than the baseline on the SIMC, while it improves by almost 4.5% on the CASMEII. 4) The largest improvements in accuracy are almost 13%, 6%, and 4.5% for the HIGO-TOP, HOG-TOP, and LBP-TOP respectively. From the results, it is safe to conclude that our model achieves superior accuracy to the current best [7], regardless of the video descriptors chosen.

B. Comparison to the state of the arts

We compare our method to nine state-of-the-art approaches, including both the deep learning method [13] and the conventional hand-crafting feature approaches [2, 3, 26, 39, 40, 33, 34, 7]. The current best method [7] provides a standard pipeline for MER tasks. As the HIGO-TOP descriptor is recommended by [7], we also use it as the video descriptors. The results are summarized in Table. 1 and Table. 2. For all the databases, our method achieves the best accuracy against all the other methods. More specifically, the accuracy on SMIC-HS, SMIC-subHS, SMIC-VIS, SMIC-NIR, and CASMEII are 68.90%, 87.32%, 83.10%, 69.01%, and 70.85% respectively, while the second best results achieved by [7] are only 68.29%, 80.28%, 81.69%, 67.61%, and 67.21% respectively. It indicates that our ME-Booster is robust to the substantial variations in appearance brought by different sensors.

C. Computational cost evaluation

Finally, we compare the computational cost of ME-Booster to the current best work [7]. The experiment is conducted on MATLAB R2017b platform, and the desktop is with an Intel CPU i5-6500 (3.20GHz) and RAM of 8 GB. Results indicate that our model is more efficient on both databases. For SMIC, the running time are 125.49 seconds (s) for Li et al. [7] and 27.42 s for our model. For CASMEII, our model can be even ten times faster than the model in [7], the running time are 420.89 s and 41.61 s, respectively.

3 CONCLUSIONS

We explore the underlying relationship between MAG and TIM and reach their common linear nature. In line with this important finding, we propose a consolidated Eulerian framework, ME-Booster, for revealing the subtle and fleeting facial expressions. ME-Booster boosts the performance of MER in both speed and accuracy. On one hand, it avoids unnecessary separation of the two modules and thus the computational cost is reduced. On the other hand, the accuracy is further improved since it eliminates the side effects brought by the intermediate process. ME-Booster is thus a distillation of the MAG and TIM, beyond a simple knockoff just putting them together. It is online, training-free, and compatible with any succeeding processing methods in the MER pipeline. Experiments on two databases indicate that our model outperforms the state-of-the-art approaches in an efficient manner.

4 ACKNOWLEDGMENTS

This work was supported by the Academy of Finland, Tekes Fidipro program (Grant No. 1849/31/2015), Business Finland project (Grant No. 3116/31/2017), and Infotech Oulu. It was also supported, in part, by the National Natural Science Foundation of China (No. 61772419 $\&$ 61572205). Moreover, the authors wish to acknowledge CSC-IT Center for Science, Finland, for computational resources. Furthermore, we express deep gratitude to the support of NVIDIA Corporation with the donation of GPUs for this research.