AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes

TL;DR

This paper introduces AMELI, a new multimodal entity linking benchmark and approach that leverages fine-grained attributes from text and images to improve entity disambiguation.

Contribution

It presents the first integration of structured attributes into multimodal entity linking and provides a comprehensive dataset and baseline models for future research.

Findings

Attributes significantly improve linking accuracy.

Multimodal data with attributes enhances disambiguation.

Proposed approach outperforms existing models.

Abstract

We propose attribute-aware multimodal entity linking, where the input consists of a mention described with a text paragraph and images, and the goal is to predict the corresponding target entity from a multimodal knowledge base (KB) where each entity is also accompanied by a text description, visual images, and a collection of attributes that present the meta-information of the entity in a structured format. To facilitate this research endeavor, we construct AMELI, encompassing a new multimodal entity linking benchmark dataset that contains 16,735 mentions described in text and associated with 30,472 images, and a multimodal knowledge base that covers 34,690 entities along with 177,873 entity images and 798,216 attributes. To establish baseline performance on AMELI, we experiment with several state-of-the-art architectures for multimodal entity linking and further propose a new approach that incorporates attributes of entities into disambiguation. Experimental results and extensive qualitative analysis demonstrate that extracting and understanding the attributes of mentions from their text descriptions and visual images play a vital role in multimodal entity linking. To the best of our knowledge, we are the first to integrate attributes in the multimodal entity linking task. The programs, model checkpoints, and the dataset are publicly available at https://github.com/VT-NLP/Ameli.