Object Segmentation from Open-Vocabulary Manipulation Instructions Based on Optimal Transport Polygon Matching with Multimodal Foundation Models

TL;DR

This paper introduces a novel method for generating object segmentation masks from open vocabulary instructions, effectively handling polygon vertex order variations and objects outside camera view, with significant performance improvements.

Contribution

The study presents a new loss function based on optimal transport for polygon matching and constructs a dataset for evaluating open vocabulary segmentation in robotic manipulation.

Findings

Achieved +16.32% improvement over existing methods.

Effectively handles polygon vertex order discrepancies.

Demonstrates robustness to objects outside camera view.

Abstract

We consider the task of generating segmentation masks for the target object from an object manipulation instruction, which allows users to give open vocabulary instructions to domestic service robots. Conventional segmentation generation approaches often fail to account for objects outside the camera's field of view and cases in which the order of vertices differs but still represents the same polygon, which leads to erroneous mask generation. In this study, we propose a novel method that generates segmentation masks from open vocabulary instructions. We implement a novel loss function using optimal transport to prevent significant loss where the order of vertices differs but still represents the same polygon. To evaluate our approach, we constructed a new dataset based on the REVERIE dataset and Matterport3D dataset. The results demonstrated the effectiveness of the proposed method compared with existing mask generation methods. Remarkably, our best model achieved a +16.32% improvement on the dataset compared with a representative polygon-based method.