Spiking Neural Networks for Early Prediction in Human Robot Collaboration

TL;DR

This paper presents TTSNet, a spiking neural network model that predicts human or agent turn-taking intentions early using multimodal cues, improving robotic collaboration efficiency.

Contribution

The paper introduces TTSNet, a novel cognitive model leveraging multimodal communication cues for early turn-taking prediction in human-robot collaboration.

Findings

TTSNet achieved an F1 score of 0.683 at 10% action completion.

It reached an F1 score of 0.852 at 50% and 0.894 at 100% completion.

TTSNet outperformed existing algorithms and surpassed human performance with limited partial observations.

Abstract

This paper introduces the Turn-Taking Spiking Neural Network (TTSNet), which is a cognitive model to perform early turn-taking prediction about human or agent's intentions. The TTSNet framework relies on implicit and explicit multimodal communication cues (physical, neurological and physiological) to be able to predict when the turn-taking event will occur in a robust and unambiguous fashion. To test the theories proposed, the TTSNet framework was implemented on an assistant robotic nurse, which predicts surgeon's turn-taking intentions and delivers surgical instruments accordingly. Experiments were conducted to evaluate TTSNet's performance in early turn-taking prediction. It was found to reach a F1 score of 0.683 given 10% of completed action, and a F1 score of 0.852 at 50% and 0.894 at 100% of the completed action. This performance outperformed multiple state-of-the-art algorithms, and surpassed human performance when limited partial observation is given (< 40%). Such early turn-taking prediction capability would allow robots to perform collaborative actions proactively, in order to facilitate collaboration and increase team efficiency.