Transcribing Natural Languages for The Deaf via Neural Editing Programs

TL;DR

This paper introduces a neural editing program approach for transcribing spoken language into sign language glosses, improving accuracy over previous sequence-to-sequence models by learning to execute editing actions.

Contribution

It proposes a novel neural agent that learns to synthesize and execute editing programs for glossification, capturing complex language connections more effectively.

Findings

Outperforms previous models significantly

Uses editing actions like addition, deletion, copying

Achieves higher transcription quality

Abstract

This work studies the task of glossification, of which the aim is to em transcribe natural spoken language sentences for the Deaf (hard-of-hearing) community to ordered sign language glosses. Previous sequence-to-sequence language models trained with paired sentence-gloss data often fail to capture the rich connections between the two distinct languages, leading to unsatisfactory transcriptions. We observe that despite different grammars, glosses effectively simplify sentences for the ease of deaf communication, while sharing a large portion of vocabulary with sentences. This has motivated us to implement glossification by executing a collection of editing actions, e.g. word addition, deletion, and copying, called editing programs, on their natural spoken language counterparts. Specifically, we design a new neural agent that learns to synthesize and execute editing programs, conditioned on sentence contexts and partial editing results. The agent is trained to imitate minimal editing programs, while exploring more widely the program space via policy gradients to optimize sequence-wise transcription quality. Results show that our approach outperforms previous glossification models by a large margin.