Beyond the Deep Metric Learning: Enhance the Cross-Modal Matching with Adversarial Discriminative Domain Regularization

TL;DR

This paper introduces Adversarial Discriminative Domain Regularization (ADDR), a novel framework that enhances cross-modal image-text matching by regulating feature distributions, leading to significant performance improvements over existing methods.

Contribution

The paper proposes a new ADDR framework that improves cross-modal matching by considering feature distribution regularization, surpassing traditional metric learning approaches.

Findings

Significant performance gains on MS-COCO and Flickr30K benchmarks.

Improved learning efficiency and matching accuracy across multiple models.

Effective regulation of feature distributions enhances cross-modal similarity measurement.

Abstract

Matching information across image and text modalities is a fundamental challenge for many applications that involve both vision and natural language processing. The objective is to find efficient similarity metrics to compare the similarity between visual and textual information. Existing approaches mainly match the local visual objects and the sentence words in a shared space with attention mechanisms. The matching performance is still limited because the similarity computation is based on simple comparisons of the matching features, ignoring the characteristics of their distribution in the data. In this paper, we address this limitation with an efficient learning objective that considers the discriminative feature distributions between the visual objects and sentence words. Specifically, we propose a novel Adversarial Discriminative Domain Regularization (ADDR) learning framework, beyond the paradigm metric learning objective, to construct a set of discriminative data domains within each image-text pairs. Our approach can generally improve the learning efficiency and the performance of existing metrics learning frameworks by regulating the distribution of the hidden space between the matching pairs. The experimental results show that this new approach significantly improves the overall performance of several popular cross-modal matching techniques (SCAN, VSRN, BFAN) on the MS-COCO and Flickr30K benchmarks.