Physically Plausible Animation of Human Upper Body from a Single Image

TL;DR

This paper introduces a novel method for generating controllable, photorealistic human upper body animations from a single image using reinforcement learning to produce dynamic, responsive keypoint sequences that are translated into realistic videos.

Contribution

The paper proposes a reinforcement learning framework that predicts human motion from a single image, enabling controllable and physically plausible animations with photorealistic quality.

Findings

Generated animations respond to user interactions and perturbations.

Produced keypoint sequences achieve task goals effectively.

The method creates realistic videos from minimal input images.

Abstract

We present a new method for generating controllable, dynamically responsive, and photorealistic human animations. Given an image of a person, our system allows the user to generate Physically plausible Upper Body Animation (PUBA) using interaction in the image space, such as dragging their hand to various locations. We formulate a reinforcement learning problem to train a dynamic model that predicts the person's next 2D state (i.e., keypoints on the image) conditioned on a 3D action (i.e., joint torque), and a policy that outputs optimal actions to control the person to achieve desired goals. The dynamic model leverages the expressiveness of 3D simulation and the visual realism of 2D videos. PUBA generates 2D keypoint sequences that achieve task goals while being responsive to forceful perturbation. The sequences of keypoints are then translated by a pose-to-image generator to produce the final photorealistic video.