T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text

TL;DR

This paper introduces T2S-GPT, a novel dynamic vector quantization approach for sign language production from text, enabling more accurate and efficient sign language generation by adjusting encoding lengths based on information density.

Contribution

It proposes a dynamic vector quantization model that adapts encoding length to information density, improving sign language synthesis from text.

Findings

Effective sign language generation demonstrated on PHOENIX14T dataset.

New large German sign language dataset PHOENIX-News introduced.

Model performance improves with increased training data size.

Abstract

In this work, we propose a two-stage sign language production (SLP) paradigm that first encodes sign language sequences into discrete codes and then autoregressively generates sign language from text based on the learned codebook. However, existing vector quantization (VQ) methods are fixed-length encodings, overlooking the uneven information density in sign language, which leads to under-encoding of important regions and over-encoding of unimportant regions. To address this issue, we propose a novel dynamic vector quantization (DVA-VAE) model that can dynamically adjust the encoding length based on the information density in sign language to achieve accurate and compact encoding. Then, a GPT-like model learns to generate code sequences and their corresponding durations from spoken language text. Extensive experiments conducted on the PHOENIX14T dataset demonstrate the effectiveness of our proposed method. To promote sign language research, we propose a new large German sign language dataset, PHOENIX-News, which contains 486 hours of sign language videos, audio, and transcription texts.Experimental analysis on PHOENIX-News shows that the performance of our model can be further improved by increasing the size of the training data. Our project homepage is https://t2sgpt-demo.yinaoxiong.cn.