ERR@HRI 2024 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Interactions

TL;DR

The ERR@HRI 2024 challenge introduces a multimodal dataset and benchmark for detecting errors and failures in human-robot interactions, aiming to improve robot reliability and interaction quality.

Contribution

This paper presents a new multimodal dataset and challenge framework for detecting robot errors during human-robot interactions, fostering advancements in machine learning models for failure detection.

Findings

Benchmark dataset with multimodal interaction data

Evaluation metrics for error detection performance

Baseline results for multimodal failure detection models

Abstract

Despite the recent advancements in robotics and machine learning (ML), the deployment of autonomous robots in our everyday lives is still an open challenge. This is due to multiple reasons among which are their frequent mistakes, such as interrupting people or having delayed responses, as well as their limited ability to understand human speech, i.e., failure in tasks like transcribing speech to text. These mistakes may disrupt interactions and negatively influence human perception of these robots. To address this problem, robots need to have the ability to detect human-robot interaction (HRI) failures. The ERR@HRI 2024 challenge tackles this by offering a benchmark multimodal dataset of robot failures during human-robot interactions (HRI), encouraging researchers to develop and benchmark multimodal machine learning models to detect these failures. We created a dataset featuring multimodal non-verbal interaction data, including facial, speech, and pose features from video clips of interactions with a robotic coach, annotated with labels indicating the presence or absence of robot mistakes, user awkwardness, and interaction ruptures, allowing for the training and evaluation of predictive models. Challenge participants have been invited to submit their multimodal ML models for detection of robot errors and to be evaluated against various performance metrics such as accuracy, precision, recall, F1 score, with and without a margin of error reflecting the time-sensitivity of these metrics. The results of this challenge will help the research field in better understanding the robot failures in human-robot interactions and designing autonomous robots that can mitigate their own errors after successfully detecting them.