MUNIChus: Multilingual News Image Captioning Benchmark

TL;DR

This paper introduces MUNIChus, the first multilingual news image captioning benchmark with 9 languages, enabling evaluation and development of models that integrate news content with images across diverse languages.

Contribution

The paper presents MUNIChus, a novel benchmark dataset for multilingual news image captioning, including low-resource languages, and provides initial evaluations of state-of-the-art models.

Findings

News image captioning remains challenging across languages.

State-of-the-art models perform variably on the benchmark.

MUNIChus is publicly available for future research.

Abstract

The goal of news image captioning is to generate captions by integrating news article content with corresponding images, highlighting the relationship between textual context and visual elements. The majority of research on news image captioning focuses on English, primarily because datasets in other languages are scarce. To address this limitation, we create the first multilingual news image captioning benchmark, MUNIChus, comprising 9 languages, including several low-resource languages such as Sinhala and Urdu. We evaluate various state-of-the-art neural news image captioning models on MUNIChus and find that news image captioning remains challenging. We also make MUNIChus publicly available with over 20 models already benchmarked. MUNIChus opens new avenues for further advancements in developing and evaluating multilingual news image captioning models.