Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructions

TL;DR

This paper introduces the Multimodal Diffusion Segmentation Model (MDSM) that understands complex natural language instructions and generates precise segmentation masks for objects in indoor scenes, outperforming existing methods.

Contribution

The paper presents a novel two-stage diffusion-based model with crossmodal feature extraction for language-guided object segmentation in complex indoor environments.

Findings

MDSM achieved +10.13 mean IoU over baseline.

The model effectively handles complex referring expressions.

A new dataset was created for evaluation.

Abstract

In this study, we aim to develop a model that comprehends a natural language instruction (e.g., "Go to the living room and get the nearest pillow to the radio art on the wall") and generates a segmentation mask for the target everyday object. The task is challenging because it requires (1) the understanding of the referring expressions for multiple objects in the instruction, (2) the prediction of the target phrase of the sentence among the multiple phrases, and (3) the generation of pixel-wise segmentation masks rather than bounding boxes. Studies have been conducted on languagebased segmentation methods; however, they sometimes mask irrelevant regions for complex sentences. In this paper, we propose the Multimodal Diffusion Segmentation Model (MDSM), which generates a mask in the first stage and refines it in the second stage. We introduce a crossmodal parallel feature extraction mechanism and extend diffusion probabilistic models to handle crossmodal features. To validate our model, we built a new dataset based on the well-known Matterport3D and REVERIE datasets. This dataset consists of instructions with complex referring expressions accompanied by real indoor environmental images that feature various target objects, in addition to pixel-wise segmentation masks. The performance of MDSM surpassed that of the baseline method by a large margin of +10.13 mean IoU.