Semi-Supervised Spoken Language Glossification

TL;DR

This paper introduces S3LG, a semi-supervised framework for spoken language glossification that leverages large-scale monolingual data and self-training to improve translation accuracy from spoken language to sign language glosses.

Contribution

The paper presents a novel semi-supervised learning framework combining rule-based and model-based auto-annotation with consistency regularization for SLG.

Findings

Significant improvement over baseline models on public benchmarks.

Effective utilization of monolingual data enhances SLG performance.

Robustness of the framework against synthetic data noise.

Abstract

Spoken language glossification (SLG) aims to translate the spoken language text into the sign language gloss, i.e., a written record of sign language. In this work, we present a framework named $S$emi-$S$upervised $S$poken $L$anguage $G$lossification ($S^3$LG) for SLG. To tackle the bottleneck of limited parallel data in SLG, our $S^3$LG incorporates large-scale monolingual spoken language text into SLG training. The proposed framework follows the self-training structure that iteratively annotates and learns from pseudo labels. Considering the lexical similarity and syntactic difference between sign language and spoken language, our $S^3$LG adopts both the rule-based heuristic and model-based approach for auto-annotation. During training, we randomly mix these complementary synthetic datasets and mark their differences with a special token. As the synthetic data may be less quality, the $S^3$LG further leverages consistency regularization to reduce the negative impact of noise in the synthetic data. Extensive experiments are conducted on public benchmarks to demonstrate the effectiveness of the $S^3$LG. Our code is available at \url{https://github.com/yaohj11/S3LG}.