JUMPS: Joints Upsampling Method for Pose Sequences

TL;DR

JUMPS is a novel deep generative approach that enhances 2D human pose sequences by adding more joints and recovering occluded ones, improving motion representation for applications like animation and scene understanding.

Contribution

This paper introduces JUMPS, the first method to increase joint count and recover missing joints in 2D pose sequences using a GAN-based inpainting approach.

Findings

Localization accuracy of added joints matches original pose estimates

Method effectively recovers occluded or missing joints

Enhances motion representation for animation and analysis

Abstract

Human Pose Estimation is a low-level task useful forsurveillance, human action recognition, and scene understandingat large. It also offers promising perspectives for the animationof synthetic characters. For all these applications, and especiallythe latter, estimating the positions of many joints is desirablefor improved performance and realism. To this purpose, wepropose a novel method called JUMPS for increasing the numberof joints in 2D pose estimates and recovering occluded ormissing joints. We believe this is the first attempt to addressthe issue. We build on a deep generative model that combines aGenerative Adversarial Network (GAN) and an encoder. TheGAN learns the distribution of high-resolution human posesequences, the encoder maps the input low-resolution sequencesto its latent space. Inpainting is obtained by computing the latentrepresentation whose decoding by the GAN generator optimallymatches the joints locations at the input. Post-processing a 2Dpose sequence using our method provides a richer representationof the character motion. We show experimentally that thelocalization accuracy of the additional joints is on average onpar with the original pose estimates.