Gloss Attention for Gloss-free Sign Language Translation
Aoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin, Tao Jin, Zhou Zhao

TL;DR
This paper introduces gloss attention, a novel method for sign language translation that reduces dependence on gloss annotations by enabling the model to focus on semantically coherent video segments, improving translation accuracy.
Contribution
The paper proposes gloss attention, a new attention mechanism that mimics gloss benefits without requiring gloss annotations, and transfers sentence similarity knowledge to enhance sign language translation.
Findings
GASLT outperforms existing sign language translation methods.
Gloss attention effectively models semantic boundaries in videos.
Knowledge transfer improves sentence-level understanding.
Abstract
Most sign language translation (SLT) methods to date require the use of gloss annotations to provide additional supervision information, however, the acquisition of gloss is not easy. To solve this problem, we first perform an analysis of existing models to confirm how gloss annotations make SLT easier. We find that it can provide two aspects of information for the model, 1) it can help the model implicitly learn the location of semantic boundaries in continuous sign language videos, 2) it can help the model understand the sign language video globally. We then propose \emph{gloss attention}, which enables the model to keep its attention within video segments that have the same semantics locally, just as gloss helps existing models do. Furthermore, we transfer the knowledge of sentence-to-sentence similarity from the natural language model to our gloss attention SLT network (GASLT) to…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHand Gesture Recognition Systems · Hearing Impairment and Communication · Human Pose and Action Recognition
