Towards Multi-User Activity Recognition through Facilitated Training Data and Deep Learning for Human-Robot Collaboration Applications

TL;DR

This paper proposes a data collection method for multi-user activity recognition in human-robot collaboration, using merged single-user data to reduce recording effort, validated with deep learning models achieving comparable performance to group recordings.

Contribution

It introduces a novel approach of merging single-user data for training multi-user activity recognition models, easing data collection challenges in multi-party HRC scenarios.

Findings

Merged single-user data yields similar recognition performance to group data.

Deep learning models effectively recognize joint activities from merged data.

Method reduces effort in collecting multi-user activity datasets.

Abstract

Human-robot interaction (HRI) research is progressively addressing multi-party scenarios, where a robot interacts with more than one human user at the same time. Conversely, research is still at an early stage for human-robot collaboration. The use of machine learning techniques to handle such type of collaboration requires data that are less feasible to produce than in a typical HRC setup. This work outlines scenarios of concurrent tasks for non-dyadic HRC applications. Based upon these concepts, this study also proposes an alternative way of gathering data regarding multi-user activity, by collecting data related to single users and merging them in post-processing, to reduce the effort involved in producing recordings of pair settings. To validate this statement, 3D skeleton poses of activity of single users were collected and merged in pairs. After this, such datapoints were used to separately train a long short-term memory (LSTM) network and a variational autoencoder (VAE) composed of spatio-temporal graph convolutional networks (STGCN) to recognise the joint activities of the pairs of people. The results showed that it is possible to make use of data collected in this way for pair HRC settings and get similar performances compared to using training data regarding groups of users recorded under the same settings, relieving from the technical difficulties involved in producing these data. The related code and collected data are publicly available.