MR-COSMO: Visual-Text Memory Recall and Direct CrOSs-MOdal Alignment Method for Query-Driven 3D Segmentation

TL;DR

MR-COSMO introduces a novel cross-modal alignment and memory recall approach that significantly improves query-driven 3D segmentation by explicitly linking 3D point clouds with text and image data.

Contribution

It proposes a new method combining direct cross-modal alignment and a visual-text memory module for enhanced 3D segmentation performance.

Findings

Achieves state-of-the-art results on multiple 3D segmentation benchmarks.

Effectively links 3D points with text and image features.

Demonstrates superior performance over existing methods.

Abstract

The rapid advancement of vision-language models (VLMs) in 3D domains has accelerated research in text-query-guided point cloud processing, though existing methods underperform in point-level segmentation due to inadequate 3D-text alignment that limits local feature-text context linking. To address this limitation, we propose MR-COSMO, a Visual-Text Memory Recall and Direct CrOSs-MOdal Alignment Method for Query-Driven 3D Segmentation, establishing explicit alignment between 3D point clouds and text/2D image data through a dedicated direct cross-modal alignment module while implementing a visual-text memory module with specialized feature banks. This direct alignment mechanism enables precise fusion of geometric and semantic features, while the memory module employs specialized banks storing text features, visual features, and their correspondence mappings to dynamically enhance scene-specific representations via attention-based knowledge recall. Comprehensive experiments across 3D instruction, reference, and semantic segmentation benchmarks confirm state-of-the-art performance.