A Neurorobotic Experiment for Crossmodal Conflict Resolution in Complex Environments

TL;DR

This study explores how an iCub robot can resolve multisensory conflicts in complex environments by mimicking human responses, using a deep learning model trained on rich audiovisual data to improve robot sensorimotor integration.

Contribution

It introduces a neurorobotic experiment with a deep learning model that enables the iCub robot to replicate human-like crossmodal conflict resolution in complex scenarios.

Findings

Robots can mimic human responses to multisensory conflicts in real time.

Visual cues congruent with scene semantics induce stronger biases.

Deep learning enables the robot to process audiovisual cues effectively.

Abstract

Crossmodal conflict resolution is crucial for robot sensorimotor coupling through the interaction with the environment, yielding swift and robust behaviour also in noisy conditions. In this paper, we propose a neurorobotic experiment in which an iCub robot exhibits human-like responses in a complex crossmodal environment. To better understand how humans deal with multisensory conflicts, we conducted a behavioural study exposing 33 subjects to congruent and incongruent dynamic audio-visual cues. In contrast to previous studies using simplified stimuli, we designed a scenario with four animated avatars and observed that the magnitude and extension of the visual bias are related to the semantics embedded in the scene, i.e., visual cues that are congruent with environmental statistics (moving lips and vocalization) induce the strongest bias. We implement a deep learning model that processes stereophonic sound, facial features, and body motion to trigger a discrete behavioural response. After training the model, we exposed the iCub to the same experimental conditions as the human subjects, showing that the robot can replicate similar responses in real time. Our interdisciplinary work provides important insights into how crossmodal conflict resolution can be modelled in robots and introduces future research directions for the efficient combination of sensory observations with internally generated knowledge and expectations.