A Bayesian Multilingual Document Model for Zero-shot Topic Identification and Discovery

TL;DR

This paper introduces a Bayesian multilingual document model that learns language-independent embeddings with uncertainty, enabling effective zero-shot cross-lingual topic identification across 17 languages, outperforming neural network baselines on mid-resource languages.

Contribution

The paper extends BaySMM to a multilingual setting, incorporating Gaussian embeddings with uncertainty, and proposes a new evaluation scheme for zero-shot cross-lingual topic detection.

Findings

Competitive performance on 8 high-resource languages.

Outperforms neural models on 9 mid-resource languages.

Proposes a robust evaluation protocol for cross-lingual tasks.

Abstract

In this paper, we present a Bayesian multilingual document model for learning language-independent document embeddings. The model is an extension of BaySMM [Kesiraju et al 2020] to the multilingual scenario. It learns to represent the document embeddings in the form of Gaussian distributions, thereby encoding the uncertainty in its covariance. We propagate the learned uncertainties through linear classifiers that benefit zero-shot cross-lingual topic identification. Our experiments on 17 languages show that the proposed multilingual Bayesian document model performs competitively, when compared to other systems based on large-scale neural networks (LASER, XLM-R, mUSE) on 8 high-resource languages, and outperforms these systems on 9 mid-resource languages. We revisit cross-lingual topic identification in zero-shot settings by taking a deeper dive into current datasets, baseline systems and the languages covered. We identify shortcomings in the existing evaluation protocol (MLDoc dataset), and propose a robust alternative scheme, while also extending the cross-lingual experimental setup to 17 languages. Finally, we consolidate the observations from all our experiments, and discuss points that can potentially benefit the future research works in applications relying on cross-lingual transfers.