Hierarchical Motion Understanding via Motion Programs

TL;DR

This paper introduces Motion Programs, a neuro-symbolic representation capturing high-level motion primitives to improve human motion analysis, video synthesis, and editing, outperforming existing models in various tasks.

Contribution

It proposes a novel hierarchical, program-like representation for human motion that captures semantic primitives and demonstrates its effectiveness in analysis, synthesis, and editing tasks.

Findings

Motion Programs accurately describe diverse human motions.

Inferred programs contain meaningful primitives like arm swings.

The approach improves video interpolation and prediction performance.

Abstract

Current approaches to video analysis of human motion focus on raw pixels or keypoints as the basic units of reasoning. We posit that adding higher-level motion primitives, which can capture natural coarser units of motion such as backswing or follow-through, can be used to improve downstream analysis tasks. This higher level of abstraction can also capture key features, such as loops of repeated primitives, that are currently inaccessible at lower levels of representation. We therefore introduce Motion Programs, a neuro-symbolic, program-like representation that expresses motions as a composition of high-level primitives. We also present a system for automatically inducing motion programs from videos of human motion and for leveraging motion programs in video synthesis. Experiments show that motion programs can accurately describe a diverse set of human motions and the inferred programs contain semantically meaningful motion primitives, such as arm swings and jumping jacks. Our representation also benefits downstream tasks such as video interpolation and video prediction and outperforms off-the-shelf models. We further demonstrate how these programs can detect diverse kinds of repetitive motion and facilitate interactive video editing.