Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN

TL;DR

This paper introduces a novel approach for directly transferring prosody from sign language to speech synthesis, utilizing a GAN-based framework and a new model architecture to improve naturalness.

Contribution

It proposes SignRecGAN and S2PFormer, enabling sign-to-speech prosody transfer without requiring costly cross-modal annotations.

Findings

Synthesized speech reflects sign language emotional nuances.

The method outperforms baseline models in prosody transfer fidelity.

Code will be publicly available upon acceptance.

Abstract

Deep learning models have improved sign language-to-text translation and made it easier for non-signers to understand signed messages. When the goal is spoken communication, a naive approach is to convert signed messages into text and then synthesize speech via Text-to-Speech (TTS). However, this two-stage pipeline inevitably treat text as a bottleneck representation, causing the loss of rich non-verbal information originally conveyed in the signing. To address this limitation, we propose a novel task, \emph{Sign-to-Speech Prosody Transfer}, which aims to capture the global prosodic nuances expressed in sign language and directly integrate them into synthesized speech. A major challenge is that aligning sign and speech requires expert knowledge, making annotation extremely costly and preventing the construction of large parallel corpora. To overcome this, we introduce \emph{SignRecGAN}, a scalable training framework that leverages unimodal datasets without cross-modal annotations through adversarial learning and reconstruction losses. Furthermore, we propose \emph{S2PFormer}, a new model architecture that preserves the expressive power of existing TTS models while enabling the injection of sign-derived prosody into the synthesized speech. Extensive experiments demonstrate that the proposed method can synthesize speech that faithfully reflects the emotional content of sign language, thereby opening new possibilities for more natural sign language communication. Our code will be available upon acceptance.