Disambiguating Affective Stimulus Associations for Robot Perception and Dialogue

TL;DR

This paper presents a method for a social robot to learn and disambiguate emotional associations between sounds and expressions, enhancing natural human-robot interaction by understanding affective cues.

Contribution

The novel contribution is enabling the NICO robot to learn and model affective associations between auditory stimuli and emotional expressions using multimodal evaluation.

Findings

NICO can learn affective associations for individual subjects and stimuli.

The robot uses affective information to decide when to extend conversations.

NICO effectively assesses subject enjoyment in real HRI scenarios.

Abstract

Effectively recognising and applying emotions to interactions is a highly desirable trait for social robots. Implicitly understanding how subjects experience different kinds of actions and objects in the world is crucial for natural HRI interactions, with the possibility to perform positive actions and avoid negative actions. In this paper, we utilize the NICO robot's appearance and capabilities to give the NICO the ability to model a coherent affective association between a perceived auditory stimulus and a temporally asynchronous emotion expression. This is done by combining evaluations of emotional valence from vision and language. NICO uses this information to make decisions about when to extend conversations in order to accrue more affective information if the representation of the association is not coherent. Our primary contribution is providing a NICO robot with the ability to learn the affective associations between a perceived auditory stimulus and an emotional expression. NICO is able to do this for both individual subjects and specific stimuli, with the aid of an emotion-driven dialogue system that rectifies emotional expression incoherences. The robot is then able to use this information to determine a subject's enjoyment of perceived auditory stimuli in a real HRI scenario.