TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions

TL;DR

TCNet is a hybrid model for continuous sign language recognition that captures long-range spatio-temporal interactions efficiently using trajectories and correlated regions, achieving state-of-the-art results.

Contribution

The paper introduces TCNet, a novel hybrid network with trajectory and correlation modules that reduce computation and improve recognition accuracy in CSL.

Findings

Achieves state-of-the-art performance on four large-scale CSL datasets.

Improves word error rate by 1.5% on PHOENIX14.

Reduces computational cost through dynamic attention mechanisms.

Abstract

A key challenge in continuous sign language recognition (CSLR) is to efficiently capture long-range spatial interactions over time from the video input. To address this challenge, we propose TCNet, a hybrid network that effectively models spatio-temporal information from Trajectories and Correlated regions. TCNet's trajectory module transforms frames into aligned trajectories composed of continuous visual tokens. In addition, for a query token, self-attention is learned along the trajectory. As such, our network can also focus on fine-grained spatio-temporal patterns, such as finger movements, of a specific region in motion. TCNet's correlation module uses a novel dynamic attention mechanism that filters out irrelevant frame regions. Additionally, it assigns dynamic key-value tokens from correlated regions to each query. Both innovations significantly reduce the computation cost and memory. We perform experiments on four large-scale datasets: PHOENIX14, PHOENIX14-T, CSL, and CSL-Daily, respectively. Our results demonstrate that TCNet consistently achieves state-of-the-art performance. For example, we improve over the previous state-of-the-art by 1.5% and 1.0% word error rate on PHOENIX14 and PHOENIX14-T, respectively.