Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting

TL;DR

This paper presents a neural rendering pipeline that accurately transfers facial expressions, head pose, and body movements in sign language videos, enabling realistic sign language reenactment and anonymization.

Contribution

It introduces a novel deep learning method for photorealistic sign language retargeting, improving realism and versatility over existing approaches.

Findings

Achieves high-quality, photorealistic sign language transfer

Demonstrates effectiveness in sign language anonymization and synthesis

Outperforms previous methods in qualitative and quantitative evaluations

Abstract

In this paper, we introduce a neural rendering pipeline for transferring the facial expressions, head pose, and body movements of one person in a source video to another in a target video. We apply our method to the challenging case of Sign Language videos: given a source video of a sign language user, we can faithfully transfer the performed manual (e.g., handshape, palm orientation, movement, location) and non-manual (e.g., eye gaze, facial expressions, mouth patterns, head, and body movements) signs to a target video in a photo-realistic manner. Our method can be used for Sign Language Anonymization, Sign Language Production (synthesis module), as well as for reenacting other types of full body activities (dancing, acting performance, exercising, etc.). We conduct detailed qualitative and quantitative evaluations and comparisons, which demonstrate the particularly promising and realistic results that we obtain and the advantages of our method over existing approaches.