SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval

TL;DR

This paper introduces SEDS, a dual-stream encoder that combines pose and RGB data with attention mechanisms for improved sign language retrieval, achieving superior performance over existing methods.

Contribution

The paper proposes a novel framework integrating pose and RGB modalities with a cross-gloss attention fusion and a fine-grained matching objective for sign language retrieval.

Findings

Significantly outperforms state-of-the-art methods on multiple datasets.

Efficient end-to-end training with lightweight networks.

Effective fusion of local and global sign language features.

Abstract

Different from traditional video retrieval, sign language retrieval is more biased towards understanding the semantic information of human actions contained in video clips. Previous works typically only encode RGB videos to obtain high-level semantic features, resulting in local action details drowned in a large amount of visual information redundancy. Furthermore, existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training, and adopt offline RGB encoder instead, leading to suboptimal feature representation. To address these issues, we propose a novel sign language representation framework called Semantically Enhanced Dual-Stream Encoder (SEDS), which integrates Pose and RGB modalities to represent the local and global information of sign language videos. Specifically, the Pose encoder embeds the coordinates of keypoints corresponding to human joints, effectively capturing detailed action features. For better context-aware fusion of two video modalities, we propose a Cross Gloss Attention Fusion (CGAF) module to aggregate the adjacent clip features with similar semantic information from intra-modality and inter-modality. Moreover, a Pose-RGB Fine-grained Matching Objective is developed to enhance the aggregated fusion feature by contextual matching of fine-grained dual-stream features. Besides the offline RGB encoder, the whole framework only contains learnable lightweight networks, which can be trained end-to-end. Extensive experiments demonstrate that our framework significantly outperforms state-of-the-art methods on various datasets.