Image-Text Relation Prediction for Multilingual Tweets

TL;DR

This paper investigates how multilingual vision-language models predict the relationship between images and text in social media posts, creating a new multilingual benchmark dataset and analyzing model performance across languages.

Contribution

It introduces a new balanced multilingual Twitter dataset for image-text relation prediction and evaluates the capabilities of recent vision-language models on this task.

Findings

Recent models show improved performance but still have significant room for enhancement.

Multilingual models perform variably across languages, highlighting the need for more inclusive training.

The dataset enables fair comparison of models across different languages.

Abstract

Various social networks have been allowing media uploads for over a decade now. Still, it has not always been clear what is their relation with the posted text or even if there is any at all. In this work, we explore how multilingual vision-language models tackle the task of image-text relation prediction in different languages, and construct a dedicated balanced benchmark data set from Twitter posts in Latvian along with their manual translations into English. We compare our results to previous work and show that the more recently released vision-language model checkpoints are becoming increasingly capable at this task, but there is still much room for further improvement.