Learning Online Scale Transformation for Talking Head Video Generation

TL;DR

This paper introduces an online scale transformation module for talking head video generation that automatically adjusts the scale of the driving face to match the source, improving face reenactment accuracy.

Contribution

It proposes a novel scale transformation module integrated into the generation process, enabling automatic scale adjustment without anchor frames.

Findings

Accurately adjusts face scale in reenactment

Produces high-quality, correctly scaled face videos

Outperforms existing methods in scale consistency

Abstract

One-shot talking head video generation uses a source image and driving video to create a synthetic video where the source person's facial movements imitate those of the driving video. However, differences in scale between the source and driving images remain a challenge for face reenactment. Existing methods attempt to locate a frame in the driving video that aligns best with the source image, but imprecise alignment can result in suboptimal outcomes. To this end, we introduce a scale transformation module that can automatically adjust the scale of the driving image to fit that of the source image, by using the information of scale difference maintained in the detected keypoints of the source image and the driving frame. Furthermore, to keep perceiving the scale information of faces during the generation process, we incorporate the scale information learned from the scale transformation module into each layer of the generation process to produce a final result with an accurate scale. Our method can perform accurate motion transfer between the two images without any anchor frame, achieved through the contributions of the proposed online scale transformation facial reenactment network. Extensive experiments have demonstrated that our proposed method adjusts the scale of the driving face automatically according to the source face, and generates high-quality faces with an accurate scale in the cross-identity facial reenactment.