Do as we do: Multiple Person Video-To-Video Transfer

TL;DR

This paper introduces a marker-less method for transferring multiple people's body motions from a source to a target video, ensuring temporal consistency and preserving individual features, addressing a gap in multi-actor video transfer research.

Contribution

It presents a novel multi-person video-to-video transfer approach that handles multiple actors simultaneously and tackles identity switching issues using pose as an intermediate representation.

Findings

Successfully transfers body motion to multiple actors in target videos.

Preserves specific features like feet touching the floor and relative positions.

Achieves convincing visual quality and appearance consistency.

Abstract

Our goal is to transfer the motion of real people from a source video to a target video with realistic results. While recent advances significantly improved image-to-image translations, only few works account for body motions and temporal consistency. However, those focus only on video re-targeting for a single actor/ for single actors. In this work, we propose a marker-less approach for multiple-person video-to-video transfer using pose as an intermediate representation. Given a source video with multiple persons dancing or working out, our method transfers the body motion of all actors to a new set of actors in a different video. Differently from recent "do as I do" methods, we focus specifically on transferring multiple person at the same time and tackle the related identity switch problem. Our method is able to convincingly transfer body motion to the target video, while preserving specific features of the target video, such as feet touching the floor and relative position of the actors. The evaluation is performed with visual quality and appearance metrics using publicly available videos with the permission of their owners.