A Study on Domain Generalization for Failure Detection through Human Reactions in HRI

TL;DR

This paper analyzes the challenge of domain generalization in failure detection models for human-robot interaction, highlighting performance drops across different datasets and emphasizing the need for more robust models.

Contribution

It provides a concise analysis of domain generalization issues in failure detection models trained on human facial expressions across diverse datasets.

Findings

Models perform poorly when tested on out-of-distribution data.

Performance drops significantly across different datasets.

Highlights the need for improved robustness in HRI models.

Abstract

Machine learning models are commonly tested in-distribution (same dataset); performance almost always drops in out-of-distribution settings. For HRI research, the goal is often to develop generalized models. This makes domain generalization - retaining performance in different settings - a critical issue. In this study, we present a concise analysis of domain generalization in failure detection models trained on human facial expressions. Using two distinct datasets of humans reacting to videos where error occurs, one from a controlled lab setting and another collected online, we trained deep learning models on each dataset. When testing these models on the alternate dataset, we observed a significant performance drop. We reflect on the causes for the observed model behavior and leave recommendations. This work emphasizes the need for HRI research focusing on improving model robustness and real-life applicability.