Can images help recognize entities? A study of the role of images for Multimodal NER

TL;DR

This paper investigates the effectiveness of incorporating images and captions into multimodal named entity recognition, revealing limitations of current models and identifying scenarios where visual information is beneficial.

Contribution

It provides an in-depth analysis of multimodal fusion techniques and the impact of captions, highlighting the bottlenecks and conditions for improved MNER performance.

Findings

Adding images does not always improve MNER performance.

Captions can enrich context and benefit certain scenarios.

Existing models face bottlenecks in leveraging multimodal information.

Abstract

Multimodal named entity recognition (MNER) requires to bridge the gap between language understanding and visual context. While many multimodal neural techniques have been proposed to incorporate images into the MNER task, the model's ability to leverage multimodal interactions remains poorly understood. In this work, we conduct in-depth analyses of existing multimodal fusion techniques from different perspectives and describe the scenarios where adding information from the image does not always boost performance. We also study the use of captions as a way to enrich the context for MNER. Experiments on three datasets from popular social platforms expose the bottleneck of existing multimodal models and the situations where using captions is beneficial.