A Feature Analysis for Multimodal News Retrieval

TL;DR

This paper explores how combining visual and textual features enhances cross-lingual news retrieval across multiple domains, demonstrating that multimodal features outperform single-modality approaches.

Contribution

It introduces a comprehensive analysis of multimodal features for cross-lingual news retrieval, highlighting the effectiveness of entity overlap and geolocation embeddings.

Findings

Multimodal features improve retrieval accuracy.

Entity overlap outperforms word embeddings for text.

Geolocation embeddings outperform other visual features.

Abstract

Content-based information retrieval is based on the information contained in documents rather than using metadata such as keywords. Most information retrieval methods are either based on text or image. In this paper, we investigate the usefulness of multimodal features for cross-lingual news search in various domains: politics, health, environment, sport, and finance. To this end, we consider five feature types for image and text and compare the performance of the retrieval system using different combinations. Experimental results show that retrieval results can be improved when considering both visual and textual information. In addition, it is observed that among textual features entity overlap outperforms word embeddings, while geolocation embeddings achieve better performance among visual features in the retrieval task.