SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification

TL;DR

SAMURAI introduces a shape-aware multimodal retrieval system that combines semantic language understanding with shape priors to improve 3D object identification in complex indoor environments, especially under limited scene context.

Contribution

The paper presents a novel hybrid retrieval framework that integrates CLIP-based semantic matching with shape-guided re-ranking and a robust preprocessing pipeline for enhanced 3D object retrieval.

Findings

Achieves competitive performance on the ROOMELSA test set.

Demonstrates the effectiveness of combining shape priors with language cues.

Highlights the importance of robust mask preprocessing for retrieval accuracy.

Abstract

Retrieving 3D objects in complex indoor environments using only a masked 2D image and a natural language description presents significant challenges. The ROOMELSA challenge limits access to full 3D scene context, complicating reasoning about object appearance, geometry, and semantics. These challenges are intensified by distorted viewpoints, textureless masked regions, ambiguous language prompts, and noisy segmentation masks. To address this, we propose SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification. SAMURAI integrates CLIP-based semantic matching with shape-guided re-ranking derived from binary silhouettes of masked regions, alongside a robust majority voting strategy. A dedicated preprocessing pipeline enhances mask quality by extracting the largest connected component and removing background noise. Our hybrid retrieval framework leverages both language and shape cues, achieving competitive performance on the ROOMELSA private test set. These results highlight the importance of combining shape priors with language understanding for robust open-world 3D object retrieval.