SViP: Sequencing Bimanual Visuomotor Policies with Object-Centric Motion Primitives

TL;DR

SViP introduces a novel framework that combines visuomotor policies with task and motion planning, enabling generalization to new tasks and conditions using minimal demonstrations without relying on object pose estimation.

Contribution

The paper presents SViP, a method that integrates visuomotor policies into TAMP using scene graph-based decision variables, improving generalization and robustness in bimanual manipulation tasks.

Findings

SViP generalizes to out-of-distribution initial conditions.

Achieves successful task execution with only 20 demonstrations.

Outperforms state-of-the-art imitation learning methods.

Abstract

Imitation learning (IL), particularly when leveraging high-dimensional visual inputs for policy training, has proven intuitive and effective in complex bimanual manipulation tasks. Nonetheless, the generalization capability of visuomotor policies remains limited, especially when small demonstration datasets are available. Accumulated errors in visuomotor policies significantly hinder their ability to complete long-horizon tasks. To address these limitations, we propose SViP, a framework that seamlessly integrates visuomotor policies into task and motion planning (TAMP). SViP partitions human demonstrations into bimanual and unimanual operations using a semantic scene graph monitor. Continuous decision variables from the key scene graph are employed to train a switching condition generator. This generator produces parameterized scripted primitives that ensure reliable performance even when encountering out-of-the-distribution observations. Using only 20 real-world demonstrations, we show that SViP enables visuomotor policies to generalize across out-of-distribution initial conditions without requiring object pose estimators. For previously unseen tasks, SViP automatically discovers effective solutions to achieve the goal, leveraging constraint modeling in TAMP formulism. In real-world experiments, SViP outperforms state-of-the-art generative IL methods, indicating wider applicability for more complex tasks. Project website: https://sites.google.com/view/svip-bimanual