Learning to Manipulate Anything: Revealing Data Scaling Laws in Bounding-Box Guided Policies

TL;DR

This paper introduces a bounding-box guided diffusion policy for semantic manipulation, revealing data scaling laws that improve generalization and adaptability in real-world robotic tasks.

Contribution

It proposes a novel semantic-motion-decoupled framework and a data collection pipeline, demonstrating the existence of power-law relationships in data scaling for manipulation tasks.

Findings

Power-law relationship between generalization and number of bounding-box objects

Achieved 85% success rate across multiple tasks on unseen objects

Effective data collection strategy enhances manipulation performance

Abstract

Diffusion-based policies show limited generalization in semantic manipulation, posing a key obstacle to the deployment of real-world robots. This limitation arises because relying solely on text instructions is inadequate to direct the policy's attention toward the target object in complex and dynamic environments. To solve this problem, we propose leveraging bounding-box instruction to directly specify target object, and further investigate whether data scaling laws exist in semantic manipulation tasks. Specifically, we design a handheld segmentation device with an automated annotation pipeline, Label-UMI, which enables the efficient collection of demonstration data with semantic labels. We further propose a semantic-motion-decoupled framework that integrates object detection and bounding-box guided diffusion policy to improve generalization and adaptability in semantic manipulation. Throughout extensive real-world experiments on large-scale datasets, we validate the effectiveness of the approach, and reveal a power-law relationship between generalization performance and the number of bounding-box objects. Finally, we summarize an effective data collection strategy for semantic manipulation, which can achieve 85\% success rates across four tasks on both seen and unseen objects. All datasets and code will be released to the community.