Diversity-Aware Sign Language Production through a Pose Encoding Variational Autoencoder

TL;DR

This paper introduces a diversity-aware sign language image generation model using a pose encoding variational autoencoder, enhancing image diversity and quality while maintaining pose accuracy.

Contribution

It extends variational inference with pose and attribute conditioning, employing a UNet architecture with separate decoders for improved sign language image synthesis.

Findings

Outperforms state-of-the-art baselines in diversity and image quality

Faithfully reproduces non-manual features in sign language images

Achieves better pose estimation accuracy

Abstract

This paper addresses the problem of diversity-aware sign language production, where we want to give an image (or sequence) of a signer and produce another image with the same pose but different attributes (\textit{e.g.} gender, skin color). To this end, we extend the variational inference paradigm to include information about the pose and the conditioning of the attributes. This formulation improves the quality of the synthesised images. The generator framework is presented as a UNet architecture to ensure spatial preservation of the input pose, and we include the visual features from the variational inference to maintain control over appearance and style. We generate each body part with a separate decoder. This architecture allows the generator to deliver better overall results. Experiments on the SMILE II dataset show that the proposed model performs quantitatively better than state-of-the-art baselines regarding diversity, per-pixel image quality, and pose estimation. Quantitatively, it faithfully reproduces non-manual features for signers.