Multi-level Matching Network for Multimodal Entity Linking

TL;DR

This paper introduces M3EL, a novel multi-level matching network for multimodal entity linking that effectively captures intra- and cross-modal interactions, improving performance on multiple datasets.

Contribution

The paper proposes a multi-level matching network with intra-modal contrastive learning and bidirectional cross-modal strategies for enhanced multimodal entity linking.

Findings

Outperforms state-of-the-art baselines on WikiMEL, RichpediaMEL, and WikiDiverse datasets.

Effectively captures intra-modal differences and bidirectional cross-modal interactions.

Demonstrates significant improvements in linking accuracy across multiple datasets.

Abstract

Multimodal entity linking (MEL) aims to link ambiguous mentions within multimodal contexts to corresponding entities in a multimodal knowledge base. Most existing approaches to MEL are based on representation learning or vision-and-language pre-training mechanisms for exploring the complementary effect among multiple modalities. However, these methods suffer from two limitations. On the one hand, they overlook the possibility of considering negative samples from the same modality. On the other hand, they lack mechanisms to capture bidirectional cross-modal interaction. To address these issues, we propose a Multi-level Matching network for Multimodal Entity Linking (M3EL). Specifically, M3EL is composed of three different modules: (i) a Multimodal Feature Extraction module, which extracts modality-specific representations with a multimodal encoder and introduces an intra-modal contrastive learning sub-module to obtain better discriminative embeddings based on uni-modal differences; (ii) an Intra-modal Matching Network module, which contains two levels of matching granularity: Coarse-grained Global-to-Global and Fine-grained Global-to-Local, to achieve local and global level intra-modal interaction; (iii) a Cross-modal Matching Network module, which applies bidirectional strategies, Textual-to-Visual and Visual-to-Textual matching, to implement bidirectional cross-modal interaction. Extensive experiments conducted on WikiMEL, RichpediaMEL, and WikiDiverse datasets demonstrate the outstanding performance of M3EL when compared to the state-of-the-art baselines.