Multi-Robot Motion Planning from Vision and Language using Heat-Inspired Diffusion

TL;DR

This paper introduces LCHD, a vision-based diffusion framework that generates language-conditioned, collision-free multi-robot trajectories, improving success rates and reducing latency without explicit obstacle modeling.

Contribution

The paper presents LCHD, a novel diffusion-based approach integrating semantic priors and physical biases for language-conditioned multi-robot motion planning.

Findings

LCHD outperforms prior diffusion planners in success rate.

LCHD reduces planning latency.

LCHD handles out-of-distribution reachability scenarios effectively.

Abstract

Diffusion models have recently emerged as powerful tools for robot motion planning by capturing the multi-modal distribution of feasible trajectories. However, their extension to multi-robot settings with flexible, language-conditioned task specifications remains limited. Furthermore, current diffusion-based approaches incur high computational cost during inference and struggle with generalization because they require explicit construction of environment representations and lack mechanisms for reasoning about geometric reachability. To address these limitations, we present Language-Conditioned Heat-Inspired Diffusion (LCHD), an end-to-end vision-based framework that generates language-conditioned, collision-free trajectories. LCHD integrates CLIP-based semantic priors with a collision-avoiding diffusion kernel serving as a physical inductive bias that enables the planner to interpret language commands strictly within the reachable workspace. This naturally handles out-of-distribution scenarios -- in terms of reachability -- by guiding robots toward accessible alternatives that match the semantic intent, while eliminating the need for explicit obstacle information at inference time. Extensive evaluations on diverse real-world-inspired maps, along with real-robot experiments, show that LCHD consistently outperforms prior diffusion-based planners in success rate, while reducing planning latency.