Data Leakage and Evaluation Issues in Micro-Expression Analysis

TL;DR

This paper highlights data leakage and evaluation issues in micro-expression analysis, proposing standardized protocols and an open-source library to improve research reliability and reproducibility.

Contribution

It identifies common evaluation pitfalls, introduces a new standardized protocol using facial action units, and provides an open-source implementation for consistent assessment.

Findings

Data leakage significantly inflates model performance.

Fixing data leaks reduces performance to near random levels.

A new standardized evaluation protocol improves reproducibility.

Abstract

Micro-expressions have drawn increasing interest lately due to various potential applications. The task is, however, difficult as it incorporates many challenges from the fields of computer vision, machine learning and emotional sciences. Due to the spontaneous and subtle characteristics of micro-expressions, the available training and testing data are limited, which make evaluation complex. We show that data leakage and fragmented evaluation protocols are issues among the micro-expression literature. We find that fixing data leaks can drastically reduce model performance, in some cases even making the models perform similarly to a random classifier. To this end, we go through common pitfalls, propose a new standardized evaluation protocol using facial action units with over 2000 micro-expression samples, and provide an open source library that implements the evaluation protocols in a standardized manner. Code is publicly available in \url{https://github.com/tvaranka/meb}.