Pose-Guided Fine-Grained Sign Language Video Generation

TL;DR

This paper introduces a novel pose-guided motion model for generating high-quality, temporally consistent sign language videos, addressing issues of detail distortion and flickering present in previous methods.

Contribution

The paper presents a new Coarse Motion Module and Pose Fusion Module that improve detail accuracy and temporal consistency in sign language video synthesis.

Findings

Outperforms state-of-the-art methods in benchmark tests.

Shows significant improvements in detail preservation.

Achieves better temporal consistency in generated videos.

Abstract

Sign language videos are an important medium for spreading and learning sign language. However, most existing human image synthesis methods produce sign language images with details that are distorted, blurred, or structurally incorrect. They also produce sign language video frames with poor temporal consistency, with anomalies such as flickering and abrupt detail changes between the previous and next frames. To address these limitations, we propose a novel Pose-Guided Motion Model (PGMM) for generating fine-grained and motion-consistent sign language videos. Firstly, we propose a new Coarse Motion Module (CMM), which completes the deformation of features by optical flow warping, thus transfering the motion of coarse-grained structures without changing the appearance; Secondly, we propose a new Pose Fusion Module (PFM), which guides the modal fusion of RGB and pose features, thus completing the fine-grained generation. Finally, we design a new metric, Temporal Consistency Difference (TCD) to quantitatively assess the degree of temporal consistency of a video by comparing the difference between the frames of the reconstructed video and the previous and next frames of the target video. Extensive qualitative and quantitative experiments show that our method outperforms state-of-the-art methods in most benchmark tests, with visible improvements in details and temporal consistency.