Motion Transformer for Unsupervised Image Animation

TL;DR

This paper introduces a novel motion transformer model for unsupervised image animation, leveraging vision transformers to better model motion interactions and improve animation quality.

Contribution

It is the first to apply vision transformers for motion estimation in image animation, explicitly modeling interactions between motion features.

Findings

Achieves state-of-the-art results on benchmark datasets.

Effectively models motion interactions with multi-head self attention.

Improves animation quality compared to CNN-based methods.

Abstract

Image animation aims to animate a source image by using motion learned from a driving video. Current state-of-the-art methods typically use convolutional neural networks (CNNs) to predict motion information, such as motion keypoints and corresponding local transformations. However, these CNN based methods do not explicitly model the interactions between motions; as a result, the important underlying motion relationship may be neglected, which can potentially lead to noticeable artifacts being produced in the generated animation video. To this end, we propose a new method, the motion transformer, which is the first attempt to build a motion estimator based on a vision transformer. More specifically, we introduce two types of tokens in our proposed method: i) image tokens formed from patch features and corresponding position encoding; and ii) motion tokens encoded with motion information. Both types of tokens are sent into vision transformers to promote underlying interactions between them through multi-head self attention blocks. By adopting this process, the motion information can be better learned to boost the model performance. The final embedded motion tokens are then used to predict the corresponding motion keypoints and local transformations. Extensive experiments on benchmark datasets show that our proposed method achieves promising results to the state-of-the-art baselines. Our source code will be public available.