Latent Adaptive Planner for Dynamic Manipulation

TL;DR

The paper introduces the Latent Adaptive Planner (LAP), a novel approach for dynamic manipulation that learns from human videos, enabling real-time adaptation and transfer across robots in tasks like box catching.

Contribution

LAP is a trajectory-level latent-variable policy that formulates planning as inference, incorporating a model-based mapping to bridge human-robot embodiment gaps, and demonstrates superior performance in dynamic tasks.

Findings

Higher success rates in box catching tasks

More trajectory smoothness and energy efficiency

Effective transfer across different robot platforms

Abstract

We present the Latent Adaptive Planner (LAP), a trajectory-level latent-variable policy for dynamic nonprehensile manipulation (e.g., box catching) that formulates planning as inference in a low-dimensional latent space and is learned effectively from human demonstration videos. During execution, LAP achieves real-time adaptation by maintaining a posterior over the latent plan and performing variational replanning as new observations arrive. To bridge the embodiment gap between humans and robots, we introduce a model-based proportional mapping that regenerates accurate kinematic-dynamic joint states and object positions from human demonstrations. Through challenging box catching experiments with varying object properties, LAP demonstrates superior success rates, trajectory smoothness, and energy efficiency by learning human-like compliant motions and adaptive behaviors. Overall, LAP enables dynamic manipulation with real-time adaptation and successfully transfer across heterogeneous robot platforms using the same human demonstration videos.