End-to-end Semantic Object Detection with Cross-Modal Alignment

TL;DR

This paper introduces an end-to-end model for semantic object detection that aligns object proposals with text queries using contrastive learning, improving semantic image search by localizing relevant objects within images.

Contribution

It extends existing detection models with a cross-modal alignment approach, enabling end-to-end training for more accurate semantic object search.

Findings

Effective proposal-text alignment via contrastive learning

End-to-end training improves semantic search accuracy

Model localizes relevant objects within images based on text queries

Abstract

Traditional semantic image search methods aim to retrieve images that match the meaning of the text query. However, these methods typically search for objects on the whole image, without considering the localization of objects within the image. This paper presents an extension of existing object detection models for semantic image search that considers the semantic alignment between object proposals and text queries, with a focus on searching for objects within images. The proposed model uses a single feature extractor, a pre-trained Convolutional Neural Network, and a transformer encoder to encode the text query. Proposal-text alignment is performed using contrastive learning, producing a score for each proposal that reflects its semantic alignment with the text query. The Region Proposal Network (RPN) is used to generate object proposals, and the end-to-end training process allows for an efficient and effective solution for semantic image search. The proposed model was trained end-to-end, providing a promising solution for semantic image search that retrieves images that match the meaning of the text query and generates semantically relevant object proposals.