Iterative Shaping of Multi-Particle Aggregates based on Action Trees and VLM

TL;DR

This paper presents a novel robotic manipulation approach that combines vision language models and Fourier series-based shape representation to autonomously shape and transport multi-particle aggregates.

Contribution

It introduces a new method integrating VLMs for task planning and Fourier series for shape control in multi-particle manipulation tasks.

Findings

Effective autonomous shaping of particle aggregates demonstrated

High system cohesion maintained during manipulation

Trajectory adaptation based on aggregate geometry

Abstract

In this paper, we address the problem of manipulating multi-particle aggregates using a bimanual robotic system. Our approach enables the autonomous transport of dispersed particles through a series of shaping and pushing actions using robotically-controlled tools. Achieving this advanced manipulation capability presents two key challenges: high-level task planning and trajectory execution. For task planning, we leverage Vision Language Models (VLMs) to enable primitive actions such as tool affordance grasping and non-prehensile particle pushing. For trajectory execution, we represent the evolving particle aggregate's contour using truncated Fourier series, providing efficient parametrization of its closed shape. We adaptively compute trajectory waypoints based on group cohesion and the geometric centroid of the aggregate, accounting for its spatial distribution and collective motion. Through real-world experiments, we demonstrate the effectiveness of our methodology in actively shaping and manipulating multi-particle aggregates while maintaining high system cohesion.