DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking

TL;DR

This paper introduces DRIN, a novel dynamic network that models fine-grained, relation-specific alignments between mentions and entities in multimodal contexts, significantly improving MEL performance.

Contribution

The paper proposes a dynamic GCN-based framework that explicitly models multiple alignment types and adaptively selects relations for better multimodal entity linking.

Findings

DRIN outperforms state-of-the-art methods on two datasets.

Explicit relation modeling improves alignment accuracy.

Dynamic selection enhances performance on complex data.

Abstract

Multimodal Entity Linking (MEL) is a task that aims to link ambiguous mentions within multimodal contexts to referential entities in a multimodal knowledge base. Recent methods for MEL adopt a common framework: they first interact and fuse the text and image to obtain representations of the mention and entity respectively, and then compute the similarity between them to predict the correct entity. However, these methods still suffer from two limitations: first, as they fuse the features of text and image before matching, they cannot fully exploit the fine-grained alignment relations between the mention and entity. Second, their alignment is static, leading to low performance when dealing with complex and diverse data. To address these issues, we propose a novel framework called Dynamic Relation Interactive Network (DRIN) for MEL tasks. DRIN explicitly models four different types of alignment between a mention and entity and builds a dynamic Graph Convolutional Network (GCN) to dynamically select the corresponding alignment relations for different input samples. Experiments on two datasets show that DRIN outperforms state-of-the-art methods by a large margin, demonstrating the effectiveness of our approach.