MultiMediate'24: Multi-Domain Engagement Estimation

TL;DR

This paper introduces the MultiMediate'24 challenge, focusing on multi-domain engagement estimation across different languages, cultures, group sizes, and interaction types, highlighting the need for models to generalize beyond single-domain scenarios.

Contribution

It presents the first multi-domain engagement estimation challenge using diverse datasets, emphasizing cross-domain generalization and providing baseline results and solutions.

Findings

Baseline models show limited cross-domain generalization.

Multi-domain data improves robustness of engagement estimation.

Different modalities and domains pose unique challenges for models.

Abstract

Estimating the momentary level of participant's engagement is an important prerequisite for assistive systems that support human interactions. Previous work has addressed this task in within-domain evaluation scenarios, i.e. training and testing on the same dataset. This is in contrast to real-life scenarios where domain shifts between training and testing data frequently occur. With MultiMediate'24, we present the first challenge addressing multi-domain engagement estimation. As training data, we utilise the NOXI database of dyadic novice-expert interactions. In addition to within-domain test data, we add two new test domains. First, we introduce recordings following the NOXI protocol but covering languages that are not present in the NOXI training data. Second, we collected novel engagement annotations on the MPIIGroupInteraction dataset which consists of group discussions between three to four people. In this way, MultiMediate'24 evaluates the ability of approaches to generalise across factors such as language and cultural background, group size, task, and screen-mediated vs. face-to-face interaction. This paper describes the MultiMediate'24 challenge and presents baseline results. In addition, we discuss selected challenge solutions.