Spotting Micro-Expressions on Long Videos Sequences

TL;DR

This paper introduces baseline results for micro-expression spotting in long videos using local temporal patterns and PCA, evaluated on SAMM and CAS(ME)2 datasets, with a focus on defining true positives and F1-score metrics.

Contribution

It presents a novel baseline approach for micro-expression spotting using local temporal patterns and PCA, setting evaluation criteria and metrics for future research.

Findings

Baseline F1-score of 0.0316 on SAMM

Baseline F1-score of 0.0179 on CAS(ME)2

Evaluation framework with true positive criteria and F1-score metric

Abstract

This paper presents baseline results for the first Micro-Expression Spotting Challenge 2019 by evaluating local temporal pattern (LTP) on SAMM and CAS(ME)2. The proposed LTP patterns are extracted by applying PCA in a temporal window on several facial local regions. The micro-expression sequences are then spotted by a local classification of LTP and a global fusion. The performance is evaluated by Leave-One-Subject-Out cross validation. Furthermore, we define the criteria of determining true positives in one video by overlap rate and set the metric F1-score for spotting performance of the whole database. The F1-score of baseline results for SAMM and CAS(ME)2 are 0.0316 and 0.0179, respectively.

Tables

Table 1. TABLE I: A Comparison between SAMM and CAS(ME) 2 .

Database	Participants	Samples	Resolution	FPS
SAMM	32	79	2040$\times$1088	200
CAS(ME)2	22	97	640$\times$480	30

Table 2. TABLE II: Baseline result for micro-expression spotting. SAMM c M ​ E superscript subscript absent 𝑀 𝐸 𝑐 {}_{ME}^{c} represents the SAMM cropped-face videos contain ME, SAMM f M ​ E superscript subscript absent 𝑀 𝐸 𝑓 {}_{ME}^{f} are the ME videos with full frame, CAS(ME) M ​ E 2 subscript superscript absent 2 𝑀 𝐸 {}^{2}_{ME} means all the videos in this sub-dataset of CAS(ME) 2 have ME sequences.

Method	LTP-ML				LBP-${\chi }^2$
database	SAMM${}_{}{}^c{}_{ME}{}^{}$	SAMM${}_{}{}^f{}_{ME}{}^{}$	CAS(ME)${}_{ME}{}^{}{}_{}{}^2$	CAS(ME)2	SAMM${}_{}{}^c{}_{ME}{}^{}$	CAS(ME)${}_{ME}{}^{}{}_{}{}^2$	CAS(ME)2
nb_vid	79	79	32	97	79	32	97
TP	34	47	16	16	12	10	10
FP	1958	3891	1711	5742	4172	1729	5435
FN	125	112	41	41	147	47	47
Precision	0.0171	0.0043	0.0093	0.0028	0.0028	0.0057	0.0018
Recall	0.2138	0.2956	0.2807	0.2807	0.0755	0.1754	0.1754
F1-score	0.0316	0.0229	0.0179	0.0055	0.0055	0.0111	0.0035

Table 3. TABLE III: Parameter configuration for SAMM and CAS(ME) 2 . L w ​ i ​ n ​ d ​ o ​ w subscript 𝐿 𝑤 𝑖 𝑛 𝑑 𝑜 𝑤 L_{window} is the length of sliding window W v ​ i ​ d ​ e ​ o subscript 𝑊 𝑣 𝑖 𝑑 𝑒 𝑜 W_{video} , L o ​ v ​ e ​ r ​ l ​ a ​ p subscript 𝐿 𝑜 𝑣 𝑒 𝑟 𝑙 𝑎 𝑝 L_{overlap} is the overlap size between sliding windows, L i ​ n ​ t ​ e ​ r ​ v ​ a ​ l subscript 𝐿 𝑖 𝑛 𝑡 𝑒 𝑟 𝑣 𝑎 𝑙 L_{interval} is the interval length of W R ​ O ​ I subscript 𝑊 𝑅 𝑂 𝐼 W_{ROI} .

Database	$L_{window}$	$L_{overlap}$	$L_{interval}$	$siz{e}_{ROI}$
SAMM	200	60	60	15
CAS(ME)2	30	9	9	10

Equations

$$\left[\begin{array}{ccc}{P}_1^{m,j}(x)& \cdots & P_N^{m,j}(x)\\ P_1^{m,j}(y)& \cdots & P_N^{m,j}(y)\end{array}\right]=\Phi \times (\left[\begin{array}{ccc}{F}_1^{m,j}(1)& \cdots & F_N^{m,j}(1)\\ & \ddots & \\ F_1^{m,j}(a^2)& \cdots & F_N^{m,j}(a^2)\end{array}\right]-\bar{I})$$

$$\frac{W_{spotted}\cap W_{groundTruth}}{W_{spotted}\cup W_{groundTruth}}\ge k$$

$$Recall=\frac{a}{m},Precision=\frac{a}{n}$$

$$F-score=\frac{2TP}{2TP+FP+FN}=\frac{2a}{m+n}$$

$$Recal{l}_D=\frac{{\sum }_{i=1}^V{a}_i}{{\sum }_{i=1}^V{m}_i}=\frac{A}{M}$$

$$Precisio{n}_D=\frac{{\sum }_{i=1}^V{a}_i}{{\sum }_{i=1}^V{n}_i}=\frac{A}{N}$$

$$F1-scor{e}_D=\frac{2\times (Recal{l}_D\times Precisio{n}_D)}{Recal{l}_D+Precisio{n}_D}$$

Full text

††publicationid: pubid: 978-1-7281-0089-0/19/$31.00 ©2019 IEEE

I INTRODUCTION

Facial micro-expression (ME) is a local brief facial movement, which can be triggered under high emotional pressure. The duration is less than 500ms [1]. It is a very important non-verbal communication clue, the involuntary nature make it possible to analyze personal genuine emotional state. ME analysis has many potential applications in national security [2], medical care [3], educational psychology [4], and political psychology [5]. Due to the growth and importance of MEs, researchers [6] have worked collaboratively to solicit the works in this area by conducting challenges in datasets and methods for MEs. This year, the theme of the Second Facial Micro-Expression Grand Challenge has extended to spotting challenges.

The main idea of most methods for ME spotting is to compare the feature differences between the first frame and the other frames in a time window. Meanwhile, the feature descriptors used in the state of the art are diverse, to name a few: LBP [7, 8], HOG [9], optical flow [10, 11, 12, 13], integral projection [14], Riesz pyramid [15], and frequency domain [16]. Feature differences allow consistent comparisons between frames over a time window of the size of an ME. However, the movements spotted between frames might not be the ME movements, it could be noises, macro-movements and illumination changes. This is why the ability to distinguish MEs from other movements (such as blinking or subtle head movements) remains an open challenge.

Nowadays, methods utilizing machine learning are emerging [17, 18, 19, 20]. Furthermore, [21] employed deep learning for the first time to perform the ME spotting. The machine learning process enhances the ability of distinguishing micro-expression from others. However, the spatial patterns are still the primary feature for the classifier. The temporal variation pattern of facial movement in a ME duration has yet to attract sufficient attention. Meanwhile, few articles spotted micro-expression directly from local region. However, the characteristic of that the micro-expression is a local facial movement could help to reduce the false positives.

In this paper, we spot the micro-expression clips in two recently published databases, and establish the baseline method for ME spotting challenge by using directly a temporal pattern extracted from local region [22]. Frames in a ME duration are taken into account to obtain a real temporal and local pattern (LTP), and then the LTPs are recognized by a classifier. Even though the spatial pattern is not studied, the spotted facial motions are differentiated by a fusion process from local to global. This method helps to improve the ability to distinguish ME from other movements. Furthermore, it allows finding the ME spatial local region and the temporal onset index of ME. We compare the results of our proposed LTP-ML method with a LBP approach - LBP-$\chi^{2}$-distance by Moilanen et al. [7].

The rest of the paper is organized as follows: Section II presents the methodology and performance metrics. Section III introduces the result and also shows the detailed experiment results. Section IV concludes the paper.

II Methodology

This section describes the benchmark databases, the proposed LTP-ML method, the state-of-the-art LBP method and the performance metrics.

II-A Databases

Two most recent long videos spontaneous micro-expression databases, SAMM [23] and CAS(ME)2 [24], are used for ME spotting challenge. Both databases contain long videos, which were recorded in the strictly controlled laboratory environment. Table I compares the differences between these two databases. The notable differences are the resolution and frame rates used in the experimental settings. These are indeed a great challenge for computer vision and machine learning community to produce a robust method worked for both databases, The detailed information of these two databases is presented in the following two subsections.

II-A1 SAMM Long Videos Database

SAMM database consists of a total of 32 subjects and each has 7 videos [23]. The average length of videos is 35.3s. The original release of SAMM consists of micro-movement clips labelled in Action Units. Recently, the authors [25] introduced objective classes and emotion classes for the database. The recognition challenge will be using the emotional classes from the database as ground truth. The spotting challenge focuses on 79 videos, each contains one/multiple micro-movements, with a total of 159 micro-movements. The index of onset, apex and offset frames of micro-movements were provided as the ground truth. The micro-movements interval is from onset frame to offset frame. In this database, all the micro-movements are labeled. Thus, the spotted frames can indicate not only ME but also other facial movements, such as eye blinks.

II-A2 CAS(ME)2 Database

In the part A of CAS(ME)2 database [24], there are 22 subjects and 97 long videos. The average duration is 148s. The facial movements are classified as macro- and micro-expressions. The video samples may contain multiple macro or micro facial expressions. The onset, apex, offset index for these expressions are given in the excel file. In addition, the eye blinks are labeled with onset and offset time.

II-B LTP-ML: Our Proposed Baseline Method

The baseline method is developed based on the proposed LTP-ML (local temporal pattern-machine learning) method in  [22]. The method is extended for long videos by employing a sliding temporal window. The main idea and the modification of LTP-ML method is presented in the following paragraphs.

II-B1 Pre-processing

As the ME is a local facial movement, we analyze ME only on a selection of regions of interest (ROIs). First of all, as shown in Figure 1, 84 facial landmarks are tracked in the video sequence by utilizing the Genfacetracker (©Dynamixyz). Then the size of ROI square $a$ is determined by the distance $L$ between the left and right inner corners of eyes: $a=(1/5)\times L$. 12 ROIs squares are chosen based on the regions where ME happens most frequently, i.e. the corner of the eyebrows and of the mouth. Two ROIs of nose region are chosen as references because the nose is the most rigid facial region.

Since the average duration of ME is around 300ms, and the subjects barely moved in one second, the long videos in these two databases are processed by a temporal sliding window $W_{video}$ whose length is 1s. The overlap is set to 300ms to avoid missing any possible ME movements. This, the video is separated into an ensemble of small sequences $[I_{1},I_{2},...,I_{M}]$ by sliding temporal window as shown in Figure 2. The positions of 12 chosen ROIs for all frames in one sequence are determined by the detected landmarks of the first frame in the window.

II-B2 Feature Extraction

In this part, local temporal patterns (LTPs) [22] are analyzed in the local region to distinguish ME from other movements. They are extracted from 12 ROIs respectively in each small sequence. Supposing in sequence $I_{m}$ ($m\leq M$), as illustrated in the lower part of Figure 2, PCA is performed on the temporal axis of each ROI sequence to conserve the principal variation at this region. The first two components of each ROI frame are used to analyze the variation pattern of local movement. The PCA process for ROI sequence $ROI^{m}_{j}$ ($j\leq 12$) in $I_{m}$ can be presented as in equation 1.

{strip}

[TABLE]

where $F_{n}^{m,j}$ represents the pixels in one ROI frame, $P_{n}^{m,j}=[P_{n}^{m,j}(x),P_{n}^{m,j}(y)]$ are the first two components of PCA, $n$ is the frame index in this ROI sequence ($n\leq N$). Hence, each frame in $ROI^{m}_{j}$ can be represented by a point $P_{n}^{m,j}$.

Then, a sliding window $W_{ROI}$ is set depending on the average duration of ME (300ms). The distances between the first frame and the other frames in this window are calculated. The window goes through each frame in the sequence $ROI^{m}_{j}$, and the distance set can be got as $[\Delta^{m}_{j}(n,n+1),\Delta^{m}_{j}(n,n+w),...,\Delta^{m}_{j}(n,n+W_{ROI}-1)]$, as shown in Figure 3.

The values of distance are then normalized for the entire $ROI^{m}_{j}$ to avoid the influence of different movement magnitude in different videos. Hence, the feature of frame n for $ROI^{m}_{j}$ can be represented as: $[CN^{m}_{j}$, $d^{m}_{j}(n,n+1),\cdots,d^{m}_{j}(n,n+W_{ROI}-1)]$, where $d^{m}_{j}(n,n+1)$ is the normalized distance value and the $CN^{m}_{j}$ is the normalization coefficient. The more detailed deduction process can be found in [22]. The feature for one ROI sequence of the entire long video is the concatenation of features of all the separated sequences.

II-B3 Local Classification

As presented in the above paragraph, one video contains 12 feature ensembles from 12 ROI. Li et al. [22] showed the LTP patterns are similar for all chosen ROIs for all kinds of ME. The patterns which can represent the ME local movements can be recognized by a local classification. A supervised classification SVM is employed with Leave-One-Subject-Out cross validation. The feature selection and label annotation are presented in [22].

II-B4 Global Fusion

After the LTPs which fit the local ME movement pattern are recognized, a global fusion is processed to eliminate the false positives concerning other movements and true negatives caused by our recognition process. As introduced in [22], there are three steps: a local qualification, a spatial fusion and a merge process.

II-C LBP-$\chi^{2}$-distance Method

This method is firstly proposed in  [7]. It is the most commonly used method for result comparison for ME spotting. Based on [7] and [18], the configuration of LBP-$\chi^{2}$ is set as follows: the entire face region is divided into 36 blocks. The overlap rates between blocks on axis X and Y are are 0.2 and 0.3 respectively. LBP features are extracted from blocks with uniform mapping. The radius r is set to $r=3$, and the number of neighboring points p is set to $p=8$. The $\chi^{2}$ distances of the each frame are computed in an $2\times L_{interval}+1$ interval.

First of all, the value of LBP-$\chi^{2}$-distance is compared in the whole long video. However, the method can barely spot any micro-expression intervals, while there are many false positives. This is due to this method spots the maximal movements in the video, and there are some larger movements than ME in both databases. Hence, the entire video is separated into a sub-video set by a sliding window, the setting is the same as the LTP-ML method. For each sub-video, the feature differences are calculated and sorted to find the maximal movement in this short interval. This gives the chance to spot more MEs which could be ignored in entire video comparison.

II-D Performance Metrics

There are three evaluation methods used to compare the performance of the spotting tasks:

1. True positive in one video definition Supposing there are $m$ micro-expressions in the video, and $n$ intervals are spotted. The result of this spotted interval $W_{spotted}$ is considered as true positive (TP) if it fits the following condition:

[TABLE]

where $k$ is set to 0.5, $W_{groundTruth}$ represents the micro-expression interval (onset-offset). Otherwise, the spotted interval is regarded as false positive (FP).

2. Result evaluation in one video Supposing the number of TP in one video is $a$ ($a\leq m$ and $a\leq n$), then FP = $n-a$, false positive (FN) = $m-a$, the Recall, Precision and F1-score are defined:

[TABLE]

[TABLE]

In practical, these metrics might not be suitable for some videos, as there exist the following situations on a single video:

• •

The test video does not have micro-expression sequences, thus, $m=0$, the denominator of recall will be zeros.

• •

The spotting method does not spot any intervals. The denominator of precision will be zeros since $n=0$.

• •

If there are two spotting methods, Method1 spots p intervals and Method2 spots q intervals, and $p\leq q$. Supposing for both methods, the number of true positive is 0, thus the metrics (recall, precision or F1-score) values both equal to zeros. However, in fact, the Method1 performs better than Method2.

Considering these situations, we propose for a single video, we record the result in terms of TP, FP and FN. For performance comparison, we produce a final calculation of other metrics for the entire database.

3. Evaluation for entire database Supposing in the entire database, there are $V$ videos and $M$ micro-expression sequences, and the method spot $N$ intervals in total. The database could be considered as one long video, thus, the metrics for entire database can be calculated by:

[TABLE]

[TABLE]

[TABLE]

The final results by different methods would be evaluated by F1-score since it considers the both recall and precision.

III Results and Discussion

As introduced in Section II, SAMM and CAS(ME)2 have different frame rates and resolution. Hence, the lengths of sliding window $W_{video}$, the overlap size, the interval length of $W_{ROI}$ and the ROIs size are different for these two databases. Table III lists the experimental parameters.

For CAS(ME)2 database, there are 97 videos, but only 32 videos contain micro-expressions. Thus, different results are given under two conditions: one is only considering 32 videos which have ME (CAS(ME)${}^{2}_{ME}$), another one is to include the entire database (all 97 videos). Since the raw videos in SAMM database are too big to download (700GB), only 79 videos (full frame: 270GB and cropped face: 11GB) were provided for the challenge. In this work, we report the results based on these two versions of SAMM database: one is the cropped videos (SAMM${}_{ME}^{c}$) provided by the authors using the method in [26], and the other one is the videos with full frame (SAMM${}_{ME}^{f}$). The spotting process is performed only on the downloaded databases.

III-A Experiments Results of LTP-ML Method

After performing the LTP-ML method on these two databases, the spotting results for whole database are listed in Table II. The F1-score for (SAMM${}_{ME}^{c}$) and CAS(ME)${}^{2}_{ME}$ are 0.0316 and 0.0179 respectively. LTP-ML performs better in SAMM${}_{ME}^{c}$ than SAMM${}_{ME}^{f}$, since the cropped-face process has already aligned the face region in the video, and reduced the influence of irrelevant movements. Concerning the spotting result of CAS(ME)2, there are more FPs because the video in this database which has no ME may contain macro-expressions.

III-B Experiments Results of LBP-$\chi^{2}$-distance (LBP-$\chi^{2}$) Method

The result is compared with LBP-$\chi^{2}$-distance (LBP-$\chi^{2}$) method. The spotting result is listed in Table II. For CAS(ME)${}^{2}_{ME}$, when the threshold for peak selection is set to 0.15, we can get the best result for LBP-$\chi^{2}$ method, the F1-score is 0.0111. Meanwhile, the highest F1-score of SAMM${}_{ME}^{c}$ is 0.0055 when the threshold is set to 0.05.

Compared with LTP-ML method, LBP-$\chi^{2}$ method is less accurate. LTP-ML method is capable of spotting the subtle movements based on the patterns which represented the temporal pattern variation of ME. Yet, the value of F1-score is low because of the large amounts of FP. Both databases contain noises and irrelevant facial movements, especially for CAS(ME)2, it is not easy to separate macro-expressions from micro-expressions based on 30fps videos. The ability of distinguishing ME from other movements still need to be enhanced.

IV CONCLUSIONS

This paper addresses the challenge in spotting ME on long videos sequences using two most recent databases, i.e. SAMM and CAS(ME)2. We proposed LTP-ML for spotting MEs and provided a set of performance metrics as the guideline for result evaluation on ME spotting. The baseline results of these two databases are provided in this paper. We demonstrate that our proposed method is better than the LBP approach in spotting MEs. Whilst the method was able to produce a reasonable amount of TPs, there are still a huge challenge lays ahead due to the large amount of FPs. Further research will focus on enhancing the ability of distinguishing ME from other facial movements to reduce FPs, including the implementation of deep learning approaches when we have sufficient data.

V ACKNOWLEDGMENTS

The authors gratefully acknowledge the contribution of the Organisers and Program Committee Members.