Multilingual and Multimodal Topic Modelling with Pretrained Embeddings

TL;DR

This paper introduces M3L-Contrast, a neural model that jointly learns from multilingual texts and images to produce aligned, shared topic representations across languages and modalities, leveraging pretrained embeddings.

Contribution

It proposes a novel multimodal multilingual neural topic model that effectively integrates text and image data using pretrained embeddings, enabling zero-shot and aligned topic inference.

Findings

Competitive in zero-shot multilingual topic prediction

Outperforms zero-shot models in multimodal data

Performs well even with unaligned embeddings

Abstract

This paper presents M3L-Contrast -- a novel multimodal multilingual (M3L) neural topic model for comparable data that maps texts from multiple languages and images into a shared topic space. Our model is trained jointly on texts and images and takes advantage of pretrained document and image embeddings to abstract the complexities between different languages and modalities. As a multilingual topic model, it produces aligned language-specific topics and as multimodal model, it infers textual representations of semantic concepts in images. We demonstrate that our model is competitive with a zero-shot topic model in predicting topic distributions for comparable multilingual data and significantly outperforms a zero-shot model in predicting topic distributions for comparable texts and images. We also show that our model performs almost as well on unaligned embeddings as it does on aligned embeddings.