Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning

TL;DR

This paper introduces SocialLDG, a multi-task learning framework that models dynamic social interactions in robots, achieving state-of-the-art results and providing insights into human decision-making processes.

Contribution

It presents a novel approach combining lexical priors and dynamic graph learning to model social interactions and internal states in robots.

Findings

Achieves state-of-the-art performance on public datasets.

Supports scalable learning of new social tasks without forgetting.

Provides insights into the influence of internal states and actions over time.

Abstract

For a robot to be called socially intelligent, it must be able to infer users internal states from their current behaviour, predict the users future behaviour, and if required, respond appropriately. In this work, we investigate how robots can be endowed with such social intelligence by modelling the dynamic relationship between user's internal states (latent) and actions (observable state). Our premise is that these states arise from the same underlying socio-cognitive process and influence each other dynamically. Drawing inspiration from theories in Cognitive Science, we propose a novel multi-task learning framework, termed as \textbf{SocialLDG} that explicitly models the dynamic relationship among the states represent as six distinct tasks. Our framework uses a language model to introduce lexical priors for each task and employs dynamic graph learning to model task affinity evolving with time. SocialLDG has three advantages: First, it achieves state-of-the-art performance on two challenging human-robot social interaction datasets available publicly. Second, it supports strong task scalability by learning new tasks seamlessly without catastrophic forgetting. Finally, benefiting from explicit modelling task affinity, it offers insights on how different interactions unfolds in time and how the internal states and observable actions influence each other in human decision making.