CTrGAN: Cycle Transformers GAN for Gait Transfer

TL;DR

CTrGAN is a novel cycle transformer GAN that effectively transfers and personalizes gait in videos, capable of generalizing to unseen sources and providing new metrics for gait transfer quality.

Contribution

The paper introduces CTrGAN, a cycle transformer GAN that transfers gait in videos with a single training, generalizes to unseen sources, and proposes new gait transfer quality metrics.

Findings

Produces over an order of magnitude more realistic gaits than existing methods

Can transfer gait from unseen sources without retraining

Introduces a detector for real vs. generated videos

Abstract

We introduce a novel approach for gait transfer from unconstrained videos in-the-wild. In contrast to motion transfer, the objective here is not to imitate the source's motions by the target, but rather to replace the walking source with the target, while transferring the target's typical gait. Our approach can be trained only once with multiple sources and is able to transfer the gait of the target from unseen sources, eliminating the need for retraining for each new source independently. Furthermore, we propose a novel metrics for gait transfer based on gait recognition models that enable to quantify the quality of the transferred gait, and show that existing techniques yield a discrepancy that can be easily detected. We introduce Cycle Transformers GAN (CTrGAN), that consist of a decoder and encoder, both Transformers, where the attention is on the temporal domain between complete images rather than the spatial domain between patches. Using a widely-used gait recognition dataset, we demonstrate that our approach is capable of producing over an order of magnitude more realistic personalized gaits than existing methods, even when used with sources that were not available during training. As part of our solution, we present a detector that determines whether a video is real or generated by our model.