RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video

TL;DR

RobotSeg is a novel foundation model and dataset that significantly improves robot segmentation accuracy in images and videos by addressing key limitations of existing models.

Contribution

It introduces a structure-enhanced memory associator, a robot prompt generator, and a label-efficient training strategy to enable automatic, structure-aware robot segmentation.

Findings

RobotSeg achieves state-of-the-art performance on robot segmentation tasks.

The VRS dataset contains over 2.8k videos and 138k frames with diverse robots.

Extensive experiments validate the effectiveness of the proposed methods.

Abstract

Accurate robot segmentation is a fundamental capability for robotic perception. It enables precise visual servoing for VLA systems, scalable robot-centric data augmentation, accurate real-to-sim transfer, and reliable safety monitoring in dynamic human-robot environments. Despite the strong capabilities of modern segmentation models, surprisingly it remains challenging to segment robots. This is due to robot embodiment diversity, appearance ambiguity, structural complexity, and rapid shape changes. Embracing these challenges, we introduce RobotSeg, a foundation model for robot segmentation in image and video. RobotSeg is built upon the versatile SAM 2 foundation model but addresses its three limitations for robot segmentation, namely the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations, by introducing a structure-enhanced memory associator, a robot prompt generator, and a label-efficient training strategy. These innovations collectively enable a structure-aware, automatic, and label-efficient solution. We further construct the video robot segmentation (VRS) dataset comprising over 2.8k videos (138k frames) with diverse robot embodiments and environments. Extensive experiments demonstrate that RobotSeg achieves state-of-the-art performance on both images and videos, establishing a strong foundation for future advances in robot perception.