Programmatic Concept Learning for Human Motion Description and Synthesis

TL;DR

This paper introduces Programmatic Motion Concepts, a hierarchical representation for human actions that facilitates description, editing, and synthesis of motion in videos, with a focus on data efficiency and broad applicability.

Contribution

The paper presents a novel hierarchical motion representation and a semi-supervised learning architecture that enables efficient human motion description and synthesis from limited data.

Findings

Outperforms established baselines in motion tasks

Effective in small data regimes

Enables interactive editing and controlled synthesis

Abstract

We introduce Programmatic Motion Concepts, a hierarchical motion representation for human actions that captures both low-level motion and high-level description as motion concepts. This representation enables human motion description, interactive editing, and controlled synthesis of novel video sequences within a single framework. We present an architecture that learns this concept representation from paired video and action sequences in a semi-supervised manner. The compactness of our representation also allows us to present a low-resource training recipe for data-efficient learning. By outperforming established baselines, especially in the small data regime, we demonstrate the efficiency and effectiveness of our framework for multiple applications.