Upgrading the Newsroom: An Automated Image Selection System for News Articles

TL;DR

This paper introduces an automated, multilingual image selection system for news articles that leverages hierarchical self-attention and char-level embeddings, improving retrieval accuracy over existing methods.

Contribution

The novel system combines multiple textual sources and multilingual data with advanced attention mechanisms to enhance news image selection accuracy.

Findings

Outperforms existing text-image retrieval methods in a multilingual news dataset.

Utilizes multiple textual sources to improve image relevance.

Shows advantages of multilingual and multi-source textual data in retrieval tasks.

Abstract

We propose an automated image selection system to assist photo editors in selecting suitable images for news articles. The system fuses multiple textual sources extracted from news articles and accepts multilingual inputs. It is equipped with char-level word embeddings to help both modeling morphologically rich languages, e.g. German, and transferring knowledge across nearby languages. The text encoder adopts a hierarchical self-attention mechanism to attend more to both keywords within a piece of text and informative components of a news article. We extensively experiment with our system on a large-scale text-image database containing multimodal multilingual news articles collected from Swiss local news media websites. The system is compared with multiple baselines with ablation studies and is shown to beat existing text-image retrieval methods in a weakly-supervised learning setting. Besides, we also offer insights on the advantage of using multiple textual sources and multilingual data.