Conditional Image-Text Embedding Networks

TL;DR

This paper introduces a novel end-to-end model for grounding phrases in images by learning multiple text-conditioned embeddings with automatic concept assignment, improving performance across several datasets.

Contribution

It proposes a concept weight branch for automatic phrase-to-embedding assignment, simplifying representations and enhancing grounding accuracy.

Findings

Achieved 4%, 3%, and 4% improvements on three datasets.

Verified effectiveness through comprehensive experiments.

Outperformed strong baseline models.

Abstract

This paper presents an approach for grounding phrases in images which jointly learns multiple text-conditioned embeddings in a single end-to-end model. In order to differentiate text phrases into semantically distinct subspaces, we propose a concept weight branch that automatically assigns phrases to embeddings, whereas prior works predefine such assignments. Our proposed solution simplifies the representation requirements for individual embeddings and allows the underrepresented concepts to take advantage of the shared representations before feeding them into concept-specific layers. Comprehensive experiments verify the effectiveness of our approach across three phrase grounding datasets, Flickr30K Entities, ReferIt Game, and Visual Genome, where we obtain a (resp.) 4%, 3%, and 4% improvement in grounding performance over a strong region-phrase embedding baseline.