CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation

TL;DR

CorrNet+ introduces a spatial-temporal correlation network that explicitly models human body trajectories across frames, significantly improving sign language recognition and translation performance while reducing computational costs.

Contribution

This paper presents CorrNet+, a novel model that explicitly captures inter-frame correlations for sign language understanding, outperforming previous methods with lower computational overhead.

Findings

Achieves state-of-the-art results on CSLR and SLT tasks.

Outperforms previous methods using pose-estimation networks or heatmaps.

Halves the computational overhead compared to CorrNet.

Abstract

In sign language, the conveyance of human body trajectories predominantly relies upon the coordinated movements of hands and facial expressions across successive frames. Despite the recent advancements of sign language understanding methods, they often solely focus on individual frames, inevitably overlooking the inter-frame correlations that are essential for effectively modeling human body trajectories. To address this limitation, this paper introduces a spatial-temporal correlation network, denoted as CorrNet+, which explicitly identifies body trajectories across multiple frames. In specific, CorrNet+ employs a correlation module and an identification module to build human body trajectories. Afterwards, a temporal attention module is followed to adaptively evaluate the contributions of different frames. The resultant features offer a holistic perspective on human body movements, facilitating a deeper understanding of sign language. As a unified model, CorrNet+ achieves new state-of-the-art performance on two extensive sign language understanding tasks, including continuous sign language recognition (CSLR) and sign language translation (SLT). Especially, CorrNet+ surpasses previous methods equipped with resource-intensive pose-estimation networks or pre-extracted heatmaps for hand and facial feature extraction. Compared with CorrNet, CorrNet+ achieves a significant performance boost across all benchmarks while halving the computational overhead. A comprehensive comparison with previous spatial-temporal reasoning methods verifies the superiority of CorrNet+. Code is available at https://github.com/hulianyuyy/CorrNet_Plus.