DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking

TL;DR

DeepMEL introduces a multi-agent framework for multimodal entity linking that effectively integrates textual and visual data, achieving state-of-the-art results through specialized roles and dynamic coordination.

Contribution

It proposes a novel multi-agent collaborative reasoning framework with role-specific agents and adaptive strategies for improved multimodal entity linking.

Findings

Achieves state-of-the-art accuracy on five benchmark datasets.

Effectively narrows the modal gap between text and images.

Demonstrates the effectiveness of specialized agents and adaptive iteration.

Abstract

Multimodal Entity Linking (MEL) aims to associate textual and visual mentions with entities in a multimodal knowledge graph. Despite its importance, current methods face challenges such as incomplete contextual information, coarse cross-modal fusion, and the difficulty of jointly large language models (LLMs) and large visual models (LVMs). To address these issues, we propose DeepMEL, a novel framework based on multi-agent collaborative reasoning, which achieves efficient alignment and disambiguation of textual and visual modalities through a role-specialized division strategy. DeepMEL integrates four specialized agents, namely Modal-Fuser, Candidate-Adapter, Entity-Clozer and Role-Orchestrator, to complete end-to-end cross-modal linking through specialized roles and dynamic coordination. DeepMEL adopts a dual-modal alignment path, and combines the fine-grained text semantics generated by the LLM with the structured image representation extracted by the LVM, significantly narrowing the modal gap. We design an adaptive iteration strategy, combines tool-based retrieval and semantic reasoning capabilities to dynamically optimize the candidate set and balance recall and precision. DeepMEL also unifies MEL tasks into a structured cloze prompt to reduce parsing complexity and enhance semantic comprehension. Extensive experiments on five public benchmark datasets demonstrate that DeepMEL achieves state-of-the-art performance, improving ACC by 1%-57%. Ablation studies verify the effectiveness of all modules.