HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality

TL;DR

HaHeAE is a self-supervised autoencoder that learns generalisable joint representations of hand and head movements in XR, outperforming existing methods and enabling new applications like movement generation and clustering.

Contribution

We introduce HaHeAE, a novel self-supervised autoencoder with graph and diffusion encoders for joint hand-head movement modeling in XR, demonstrating superior performance and new application capabilities.

Findings

Outperforms existing methods by up to 74% in reconstruction quality.

Generalises across users, activities, and environments.

Enables interpretable clustering and movement generation.

Abstract

Human hand and head movements are the most pervasive input modalities in extended reality (XR) and are significant for a wide range of applications. However, prior works on hand and head modelling in XR only explored a single modality or focused on specific applications. We present HaHeAE - a novel self-supervised method for learning generalisable joint representations of hand and head movements in XR. At the core of our method is an autoencoder (AE) that uses a graph convolutional network-based semantic encoder and a diffusion-based stochastic encoder to learn the joint semantic and stochastic representations of hand-head movements. It also features a diffusion-based decoder to reconstruct the original signals. Through extensive evaluations on three public XR datasets, we show that our method 1) significantly outperforms commonly used self-supervised methods by up to 74.0% in terms of reconstruction quality and is generalisable across users, activities, and XR environments, 2) enables new applications, including interpretable hand-head cluster identification and variable hand-head movement generation, and 3) can serve as an effective feature extractor for downstream tasks. Together, these results demonstrate the effectiveness of our method and underline the potential of self-supervised methods for jointly modelling hand-head behaviours in extended reality.