Invertible Frowns: Video-to-Video Facial Emotion Translation

TL;DR

This paper introduces Wav2Lip-Emotion, a video-to-video translation method that modifies facial emotions in videos while preserving lip sync, identity, and pose, using L1 reconstruction and pre-trained emotion objectives, with an automated evaluation approach.

Contribution

It extends lip synchronization architecture to enable emotion modification in videos, demonstrating effective emotion change with preserved lip sync and proposing an automated emotion evaluation method.

Findings

Successful emotion modification while maintaining lip sync.

Trade-off observed between emotion intensity and visual quality.

Automated evaluation aligns with human judgments.

Abstract

We present Wav2Lip-Emotion, a video-to-video translation architecture that modifies facial expressions of emotion in videos of speakers. Previous work modifies emotion in images, uses a single image to produce a video with animated emotion, or puppets facial expressions in videos with landmarks from a reference video. However, many use cases such as modifying an actor's performance in post-production, coaching individuals to be more animated speakers, or touching up emotion in a teleconference require a video-to-video translation approach. We explore a method to maintain speakers' lip movements, identity, and pose while translating their expressed emotion. Our approach extends an existing multi-modal lip synchronization architecture to modify the speaker's emotion using L1 reconstruction and pre-trained emotion objectives. We also propose a novel automated emotion evaluation approach and corroborate it with a user study. These find that we succeed in modifying emotion while maintaining lip synchronization. Visual quality is somewhat diminished, with a trade off between greater emotion modification and visual quality between model variants. Nevertheless, we demonstrate (1) that facial expressions of emotion can be modified with nothing other than L1 reconstruction and pre-trained emotion objectives and (2) that our automated emotion evaluation approach aligns with human judgements.