# Fusing Body Posture with Facial Expressions for Joint Recognition of Affect in Child-Robot Interaction

## TL;DR

This paper introduces a deep learning approach that combines body posture and facial expressions to improve affect recognition accuracy in child-robot interactions, outperforming facial-only methods.

## Contribution

It presents a novel multi-cue affect recognition method using hierarchical multi-label training, applicable to both joint and separate modality training, for enhanced emotion detection.

## Key findings

- Significantly better accuracy than facial-only methods
- Effective on child-robot interaction and adult acted emotion datasets
- Demonstrates the benefit of multi-cue fusion in affect recognition

## Abstract

In this paper we address the problem of multi-cue affect recognition in challenging scenarios such as child-robot interaction. Towards this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging child-robot interaction database of emotional expressions collected by us, as well as on the GEMEP public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines.