Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph

TL;DR

Generative Factor Chaining (GFC) introduces a modular diffusion-based planning model that constructs feasible multi-step manipulation plans by representing tasks as a factor graph, enabling effective generalization to new scenarios.

Contribution

GFC is a novel composable generative framework that models planning problems as factor graphs with diffusion models, improving multi-manipulator task planning and generalization.

Findings

Successfully solves complex bimanual manipulation tasks.

Exhibits strong generalization to unseen tasks with new objects and constraints.

Uses modular diffusion models for flexible plan generation.

Abstract

Learning to plan for multi-step, multi-manipulator tasks is notoriously difficult because of the large search space and the complex constraint satisfaction problems. We present Generative Factor Chaining~(GFC), a composable generative model for planning. GFC represents a planning problem as a spatial-temporal factor graph, where nodes represent objects and robots in the scene, spatial factors capture the distributions of valid relationships among nodes, and temporal factors represent the distributions of skill transitions. Each factor is implemented as a modular diffusion model, which are composed during inference to generate feasible long-horizon plans through bi-directional message passing. We show that GFC can solve complex bimanual manipulation tasks and exhibits strong generalization to unseen planning tasks with novel combinations of objects and constraints. More details can be found at: https://generative-fc.github.io/