On Evaluating the Adversarial Robustness of Foundation Models for Multimodal Entity Linking

TL;DR

This paper evaluates the robustness of multimodal entity linking models against visual adversarial attacks, revealing their vulnerabilities and proposing a new method that significantly enhances their resilience, especially under adversarial conditions.

Contribution

It provides the first systematic assessment of MEL models' robustness to visual adversarial attacks and introduces LLM-RetLink, a novel approach that improves anti-interference capabilities.

Findings

Current MEL models lack robustness against visual perturbations.

Contextual semantic information can mitigate adversarial effects.

LLM-RetLink improves MEL accuracy by up to 35.7%, especially under attack.

Abstract

The explosive growth of multimodal data has driven the rapid development of multimodal entity linking (MEL) models. However, existing studies have not systematically investigated the impact of visual adversarial attacks on MEL models. We conduct the first comprehensive evaluation of the robustness of mainstream MEL models under different adversarial attack scenarios, covering two core tasks: Image-to-Text (I2T) and Image+Text-to-Text (IT2T). Experimental results show that current MEL models generally lack sufficient robustness against visual perturbations. Interestingly, contextual semantic information in input can partially mitigate the impact of adversarial perturbations. Based on this insight, we propose an LLM and Retrieval-Augmented Entity Linking (LLM-RetLink), which significantly improves the model's anti-interference ability through a two-stage process: first, extracting initial entity descriptions using large vision models (LVMs), and then dynamically generating candidate descriptive sentences via web-based retrieval. Experiments on five datasets demonstrate that LLM-RetLink improves the accuracy of MEL by 0.4%-35.7%, especially showing significant advantages under adversarial conditions. This research highlights a previously unexplored facet of MEL robustness, constructs and releases the first MEL adversarial example dataset, and sets the stage for future work aimed at strengthening the resilience of multimodal systems in adversarial environments.