Revising Second Order Terms in Deep Animation Video Coding

TL;DR

This paper improves deep animation video coding by replacing Jacobian transformations with global rotations and stabilizing adversarial training, resulting in better head rotation animation and significant bitrate savings.

Contribution

It introduces a novel modification to First Order Motion Model by using global rotations and enhances training stability with normalization, improving animation quality and efficiency.

Findings

Better performance on head-rotation animations

40% to 80% bitrate reduction on P-frames

Improved visual quality with stabilized adversarial training

Abstract

First Order Motion Model is a generative model that animates human heads based on very little motion information derived from keypoints. It is a promising solution for video communication because first it operates at very low bitrate and second its computational complexity is moderate compared to other learning based video codecs. However, it has strong limitations by design. Since it generates facial animations by warping source-images, it fails to recreate videos with strong head movements. This works concentrates on one specific kind of head movements, namely head rotations. We show that replacing the Jacobian transformations in FOMM by a global rotation helps the system to perform better on items with head-rotations while saving 40% to 80% of bitrate on P-frames. Moreover, we apply state-of-the-art normalization techniques to the discriminator to stabilize the adversarial training which is essential for generating visually appealing videos. We evaluate the performance by the learned metics LPIPS and DISTS to show the success our optimizations.