Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction

TL;DR

This paper introduces a distribution-aware framework for continuous affect prediction that models annotation uncertainty using the Beta distribution, capturing variability and higher-order distributional features.

Contribution

It proposes a novel method that estimates the mean and standard deviation of annotations to recover distributional descriptors, enhancing affect prediction by modeling uncertainty.

Findings

Beta-based models closely match empirical annotation distributions.

The approach captures skewness, kurtosis, and quantiles in affective signals.

Achieves competitive performance with traditional regression methods.

Abstract

Emotion annotation is inherently subjective and cognitively demanding, producing signals that reflect diverse perceptions across annotators rather than a single ground truth. In continuous affect prediction, this variability is typically collapsed into point estimates such as the mean or median, discarding valuable information about annotator disagreement and uncertainty. In this work, we propose a distribution-aware framework that models annotation consensus using the Beta distribution. Instead of predicting a single affect value, models estimate the mean and standard deviation of the annotation distribution, which are transformed into valid Beta parameters through moment matching. This formulation enables the recovery of higher-order distributional descriptors, including skewness, kurtosis, and quantiles, in closed form. As a result, the model captures not only the central tendency of emotional perception but also variability, asymmetry, and uncertainty in annotator responses. We evaluate the proposed approach on the SEWA and RECOLA datasets using multimodal features. Experimental results show that Beta-based modelling produces predictive distributions that closely match the empirical annotator distributions while achieving competitive performance with conventional regression approaches. These findings highlight the importance of modelling annotation uncertainty in affective computing and demonstrate the potential of distribution-aware learning for subjective signal analysis.