Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linking

TL;DR

This paper introduces MSR-MEL, a novel unsupervised multimodal entity linking framework that synthesizes and reasons over multi-perspective evidence using large language models and graph neural networks.

Contribution

It proposes a two-stage framework combining evidence synthesis and reasoning, leveraging LLMs and graph neural networks for improved unsupervised MEL performance.

Findings

MSR-MEL outperforms state-of-the-art unsupervised methods on MEL benchmarks.

The framework effectively integrates diverse evidence types for accurate entity linking.

Graph-based evidence aggregation enhances the reasoning process in MEL.

Abstract

Multimodal Entity Linking (MEL) is a fundamental task in data management that maps ambiguous mentions with diverse modalities to the multimodal entities in a knowledge base. However, most existing MEL approaches primarily focus on optimizing instance-centric features and evidence, leaving broader forms of evidence and their intricate interdependencies insufficiently explored. Motivated by the observation that human expert decision-making process relies on multi-perspective judgment, in this work, we propose MSR-MEL, a Multi-perspective Evidence Synthesis and Reasoning framework with Large Language Models (LLMs) for unsupervised MEL. Specifically, we adopt a two-stage framework: (1) Offline Multi-Perspective Evidence Synthesis constructs a comprehensive set of evidence. This includes instance-centric evidence capturing the instance-centric multimodal information of mentions and entities, group-level evidence that aggregates neighborhood information, lexical evidence based on string overlap ratio, and statistical evidence based on simple summary statistics. A core contribution of our framework is the synthesis of group-level evidence, which effectively aggregates vital neighborhood information by graph. We first construct LLM-enhanced contextualized graphs. Subsequently, different modalities are jointly aligned through an asymmetric teacher-student graph neural network. (2) Online Multi-Perspective Evidence Reasoning leverages the power of LLM as a reasoning module to analyze the correlation and semantics of the multi-perspective evidence to induce an effective ranking strategy for accurate entity linking without supervision. Extensive experiments on widely used MEL benchmarks demonstrate that MSR-MEL consistently outperforms state-of-the-art unsupervised methods. The source code of this paper was available at: https://anonymous.4open.science/r/MSR-MEL-C21E/.