Non-Autoregressive Sign Language Production via Knowledge Distillation

TL;DR

This paper introduces a novel non-autoregressive sign language production model using knowledge distillation, which improves sign pose sequence generation accuracy and addresses issues like false decoding initiation.

Contribution

The paper proposes a new NAR-SLP model with a length regulator and knowledge distillation, enhancing sign language translation quality over existing models.

Findings

Significantly outperforms existing models in gesture distance metrics.

Effectively predicts sign sequence lengths with the length regulator.

Reduces false decoding initiation through knowledge distillation.

Abstract

Sign Language Production (SLP) aims to translate expressions in spoken language into corresponding ones in sign language, such as skeleton-based sign poses or videos. Existing SLP models are either AutoRegressive (AR) or Non-Autoregressive (NAR). However, AR-SLP models suffer from regression to the mean and error propagation during decoding. NSLP-G, a NAR-based model, resolves these issues to some extent but engenders other problems. For example, it does not consider target sign lengths and suffers from false decoding initiation. We propose a novel NAR-SLP model via Knowledge Distillation (KD) to address these problems. First, we devise a length regulator to predict the end of the generated sign pose sequence. We then adopt KD, which distills spatial-linguistic features from a pre-trained pose encoder to alleviate false decoding initiation. Extensive experiments show that the proposed approach significantly outperforms existing SLP models in both Frechet Gesture Distance and Back-Translation evaluation.