$M^3EL$: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking

TL;DR

The paper introduces $M^3EL$, a large-scale multi-modal entity linking dataset covering diverse tasks and topics, and demonstrates its effectiveness in improving model performance through a new training strategy.

Contribution

It presents a novel large-scale dataset for multi-modal entity linking with diverse topics and tasks, along with a modality-augmented training strategy to enhance model generalization.

Findings

Existing models perform poorly due to limited data and coverage.

$M^3EL$ significantly improves model accuracy across tasks.

The proposed training strategy enhances multi-modal model adaptability.

Abstract

Multi-modal Entity Linking (MEL) is a fundamental component for various downstream tasks. However, existing MEL datasets suffer from small scale, scarcity of topic types and limited coverage of tasks, making them incapable of effectively enhancing the entity linking capabilities of multi-modal models. To address these obstacles, we propose a dataset construction pipeline and publish $M^3EL$, a large-scale dataset for MEL. $M^3EL$ includes 79,625 instances, covering 9 diverse multi-modal tasks, and 5 different topics. In addition, to further improve the model's adaptability to multi-modal tasks, We propose a modality-augmented training strategy. Utilizing $M^3EL$ as a corpus, train the $\textit{CLIP}_{\textit{ND}}$ model based on $\textit{CLIP} (\textit{ViT}-\textit{B}-\textit{32})$, and conduct a comparative analysis with an existing multi-modal baselines. Experimental results show that the existing models perform far below expectations (ACC of 49.4%-75.8%), After analysis, it was obtained that small dataset sizes, insufficient modality task coverage, and limited topic diversity resulted in poor generalisation of multi-modal models. Our dataset effectively addresses these issues, and the $\textit{CLIP}_{\textit{ND}}$ model fine-tuned with $M^3EL$ shows a significant improvement in accuracy, with an average improvement of 9.3% to 25% across various tasks. Our dataset is available at https://anonymous.4open.science/r/M3EL.