Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation

TL;DR

Geo-Sign introduces hyperbolic geometry-based regularization to improve skeletal representations in sign language translation, enhancing discriminative power and fine-grained motion modeling within an end-to-end framework.

Contribution

It proposes a novel hyperbolic regularization method for skeletal features, leveraging hyperbolic space properties to improve sign language translation accuracy.

Findings

Outperforms state-of-the-art RGB methods in SLT.

Enhances fine-grained motion modeling like finger articulations.

Preserves privacy and improves computational efficiency.

Abstract

Recent progress in Sign Language Translation (SLT) has focussed primarily on improving the representational capacity of large language models to incorporate Sign Language features. This work explores an alternative direction: enhancing the geometric properties of skeletal representations themselves. We propose Geo-Sign, a method that leverages the properties of hyperbolic geometry to model the hierarchical structure inherent in sign language kinematics. By projecting skeletal features derived from Spatio-Temporal Graph Convolutional Networks (ST-GCNs) into the Poincar\'e ball model, we aim to create more discriminative embeddings, particularly for fine-grained motions like finger articulations. We introduce a hyperbolic projection layer, a weighted Fr\'echet mean aggregation scheme, and a geometric contrastive loss operating directly in hyperbolic space. These components are integrated into an end-to-end translation framework as a regularisation function, to enhance the representations within the language model. This work demonstrates the potential of hyperbolic geometry to improve skeletal representations for Sign Language Translation, improving on SOTA RGB methods while preserving privacy and improving computational efficiency. Code available here: https://github.com/ed-fish/geo-sign.