Whom to Respond To? A Transformer-Based Model for Multi-Party Social Robot Interaction

TL;DR

This paper introduces a Transformer-based framework for social robots to effectively determine when and whom to respond to in multi-party interactions, enhancing naturalness and context-awareness.

Contribution

It presents a novel multi-task learning model with new loss functions and a real-world dataset for multi-party human-robot interaction.

Findings

Achieves state-of-the-art response decision accuracy

Outperforms heuristic and single-task baselines

Handles real-world complexities like gaze misalignment

Abstract

Prior human-robot interaction (HRI) research has primarily focused on single-user interactions, where robots do not need to consider the timing or recipient of their responses. However, in multi-party interactions, such as at malls and hospitals, social robots must understand the context and decide both when and to whom they should respond. In this paper, we propose a Transformer-based multi-task learning framework to improve the decision-making process of social robots, particularly in multi-user environments. Considering the characteristics of HRI, we propose two novel loss functions: one that enforces constraints on active speakers to improve scene modeling, and another that guides response selection towards utterances specifically directed at the robot. Additionally, we construct a novel multi-party HRI dataset that captures real-world complexities, such as gaze misalignment. Experimental results demonstrate that our model achieves state-of-the-art performance in respond decisions, outperforming existing heuristic-based and single-task approaches. Our findings contribute to the development of socially intelligent social robots capable of engaging in natural and context-aware multi-party interactions.