ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters

TL;DR

ControlVAE introduces a model-based framework utilizing variational autoencoders to learn and generate realistic, skill-conditioned motion controls for physics-based characters, enabling flexible reuse and high-level task adaptation.

Contribution

The paper presents a novel ControlVAE framework that learns a rich latent skill space and a control policy using a learnable world model, improving task adaptability and realism in character animation.

Findings

Effective learning of diverse motion skills

Improved downstream task performance with conditional priors

Realistic and interactive character control demonstrated

Abstract

In this paper, we introduce ControlVAE, a novel model-based framework for learning generative motion control policies based on variational autoencoders (VAE). Our framework can learn a rich and flexible latent representation of skills and a skill-conditioned generative control policy from a diverse set of unorganized motion sequences, which enables the generation of realistic human behaviors by sampling in the latent space and allows high-level control policies to reuse the learned skills to accomplish a variety of downstream tasks. In the training of ControlVAE, we employ a learnable world model to realize direct supervision of the latent space and the control policy. This world model effectively captures the unknown dynamics of the simulation system, enabling efficient model-based learning of high-level downstream tasks. We also learn a state-conditional prior distribution in the VAE-based generative control policy, which generates a skill embedding that outperforms the non-conditional priors in downstream tasks. We demonstrate the effectiveness of ControlVAE using a diverse set of tasks, which allows realistic and interactive control of the simulated characters.