ObjEmbed: Towards Universal Multimodal Object Embeddings

TL;DR

ObjEmbed introduces a novel multimodal embedding model that decomposes images into object-specific regions with semantic and spatial embeddings, enabling accurate, versatile, and efficient vision-language understanding across various tasks.

Contribution

It presents a new object-oriented multimodal embedding approach that captures semantic and spatial information, supporting diverse visual tasks with high efficiency and superior benchmark performance.

Findings

Outperforms existing models on 18 benchmarks.

Effectively handles both region-level and image-level tasks.

Provides accurate object localization and retrieval.

Abstract

Aligning objects with corresponding textual descriptions is a fundamental challenge and a realistic requirement in vision-language understanding. While recent multimodal embedding models excel at global image-text alignment, they often struggle with fine-grained alignment between image regions and specific phrases. In this work, we present ObjEmbed, a novel MLLM embedding model that decomposes the input image into multiple regional embeddings, each corresponding to an individual object, along with global embeddings. It supports a wide range of visual understanding tasks like visual grounding, local image retrieval, and global image retrieval. ObjEmbed enjoys three key properties: (1) Object-Oriented Representation: It captures both semantic and spatial aspects of objects by generating two complementary embeddings for each region: an object embedding for semantic matching and an IoU embedding that predicts localization quality. The final object matching score combines semantic similarity with the predicted IoU, enabling more accurate retrieval. (2) Versatility: It seamlessly handles both region-level and image-level tasks. (3) Efficient Encoding: All objects in an image, along with the full image, are encoded in a single forward pass for high efficiency. Superior performance on 18 diverse benchmarks demonstrates its strong semantic discrimination.