Fighting noise and imbalance in Action Unit detection problems

TL;DR

This paper introduces Robin Hood Label Smoothing (RHLS), a novel technique to improve facial Action Unit detection by mitigating noise and class imbalance, leading to better performance on benchmark datasets.

Contribution

The paper proposes RHLS, a new label smoothing method that selectively reduces confidence for the majority class to address noise and imbalance in AU detection.

Findings

RHLS improves AU detection performance on BP4D and DISFA datasets.

Applying RHLS on a multi-task baseline outperforms state-of-the-art methods.

RHLS effectively reduces the impact of noisy and imbalanced data in facial expression analysis.

Abstract

Action Unit (AU) detection aims at automatically caracterizing facial expressions with the muscular activations they involve. Its main interest is to provide a low-level face representation that can be used to assist higher level affective computing tasks learning. Yet, it is a challenging task. Indeed, the available databases display limited face variability and are imbalanced toward neutral expressions. Furthermore, as AU involve subtle face movements they are difficult to annotate so that some of the few provided datapoints may be mislabeled. In this work, we aim at exploiting label smoothing ability to mitigate noisy examples impact by reducing confidence [1]. However, applying label smoothing as it is may aggravate imbalance-based pre-existing under-confidence issue and degrade performance. To circumvent this issue, we propose Robin Hood Label Smoothing (RHLS). RHLS principle is to restrain label smoothing confidence reduction to the majority class. In that extent, it alleviates both the imbalance-based over-confidence issue and the negative impact of noisy majority class examples. From an experimental standpoint, we show that RHLS provides a free performance improvement in AU detection. In particular, by applying it on top of a modern multi-task baseline we get promising results on BP4D and outperform state-of-the-art methods on DISFA.