Improving Visual-Semantic Embedding with Adaptive Pooling and Optimization Objective

TL;DR

This paper introduces adaptive pooling and negative sample selection strategies to improve visual-semantic embedding models, leading to better retrieval performance on standard datasets.

Contribution

It proposes simple yet effective pooling and optimization strategies that outperform complex methods and enhance convergence in VSE models.

Findings

Outperforms state-of-the-art on Flickr30K and MS-COCO datasets.

Improves Recall@K metrics by at least 1.0%.

Simple pooling methods are as effective as sophisticated ones.

Abstract

Visual-Semantic Embedding (VSE) aims to learn an embedding space where related visual and semantic instances are close to each other. Recent VSE models tend to design complex structures to pool visual and semantic features into fixed-length vectors and use hard triplet loss for optimization. However, we find that: (1) combining simple pooling methods is no worse than these sophisticated methods; and (2) only considering the most difficult-to-distinguish negative sample leads to slow convergence and poor Recall@K improvement. To this end, we propose an adaptive pooling strategy that allows the model to learn how to aggregate features through a combination of simple pooling methods. We also introduce a strategy to dynamically select a group of negative samples to make the optimization converge faster and perform better. Experimental results on Flickr30K and MS-COCO demonstrate that a standard VSE using our pooling and optimization strategies outperforms current state-of-the-art systems (at least 1.0% on the metrics of recall) in image-to-text and text-to-image retrieval. Source code of our experiments is available at https://github.com/96-Zachary/vse_2ad.