Rethinking Affect Analysis: A Protocol for Ensuring Fairness and Consistency

TL;DR

This paper introduces a unified protocol for affect analysis evaluation, addressing fairness and consistency issues by standardizing database partitioning, annotations, and metrics, and providing new leaderboards for fair comparison.

Contribution

It proposes a comprehensive, standardized evaluation framework for affect analysis, including demographic annotations and a new leaderboard to promote fairer research comparisons.

Findings

Reevaluated existing methods with the new protocol.

Identified inconsistencies in previous evaluations.

Provided resources for fair affect analysis research.

Abstract

Evaluating affect analysis methods presents challenges due to inconsistencies in database partitioning and evaluation protocols, leading to unfair and biased results. Previous studies claim continuous performance improvements, but our findings challenge such assertions. Using these insights, we propose a unified protocol for database partitioning that ensures fairness and comparability. We provide detailed demographic annotations (in terms of race, gender and age), evaluation metrics, and a common framework for expression recognition, action unit detection and valence-arousal estimation. We also rerun the methods with the new protocol and introduce a new leaderboards to encourage future research in affect recognition with a fairer comparison. Our annotations, code, and pre-trained models are available on \hyperlink{https://github.com/dkollias/Fair-Consistent-Affect-Analysis}{Github}.