Enabling a Social Robot to Process Social Cues to Detect when to Help a User

TL;DR

This paper presents a multimodal architecture enabling social robots to recognize when to assist users by analyzing social cues like eye gaze and language, aiming for real-time, context-independent assistance.

Contribution

It introduces a novel multimodal fusion approach using social cues for timely assistance, adaptable across various tasks and contexts.

Findings

Effective detection of user needs in a Lego task

Minimal task-specific dependencies in the architecture

Potential for improved user experience through social cue recognition

Abstract

It is important for socially assistive robots to be able to recognize when a user needs and wants help. Such robots need to be able to recognize human needs in a real-time manner so that they can provide timely assistance. We propose an architecture that uses social cues to determine when a robot should provide assistance. Based on a multimodal fusion approach upon eye gaze and language modalities, our architecture is trained and evaluated on data collected in a robot-assisted Lego building task. By focusing on social cues, our architecture has minimal dependencies on the specifics of a given task, enabling it to be applied in many different contexts. Enabling a social robot to recognize a user's needs through social cues can help it to adapt to user behaviors and preferences, which in turn will lead to improved user experiences.