SimulSLT: End-to-End Simultaneous Sign Language Translation
Aoxiong Yin, Zhou Zhao, Jinglin Liu, Weike Jin, Meng Zhang, Xingshan, Zeng, Xiaofei He

TL;DR
SimulSLT is an innovative end-to-end model for simultaneous sign language translation that reduces latency and improves translation quality by using a novel boundary predictor and re-encode method, enabling real-time applications.
Contribution
The paper introduces SimulSLT, the first end-to-end simultaneous sign language translation model with a boundary predictor and re-encode technique for improved real-time translation.
Findings
Achieves higher BLEU scores than non-simultaneous models.
Maintains low latency suitable for real-time use.
Effective boundary prediction enhances translation accuracy.
Abstract
Sign language translation as a kind of technology with profound social significance has attracted growing researchers' interest in recent years. However, the existing sign language translation methods need to read all the videos before starting the translation, which leads to a high inference latency and also limits their application in real-life scenarios. To solve this problem, we propose SimulSLT, the first end-to-end simultaneous sign language translation model, which can translate sign language videos into target text concurrently. SimulSLT is composed of a text decoder, a boundary predictor, and a masked encoder. We 1) use the wait-k strategy for simultaneous translation. 2) design a novel boundary predictor based on the integrate-and-fire module to output the gloss boundary, which is used to model the correspondence between the sign language video and the gloss. 3) propose an…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHand Gesture Recognition Systems · Human Pose and Action Recognition · Multimodal Machine Learning Applications
