Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification

TL;DR

This paper introduces a semi-supervised document classification method that combines deep generative models with multi-lingual pretraining, effectively addressing data scarcity in low-resource languages and outperforming existing methods.

Contribution

It presents a novel pipeline integrating deep generative models with multi-lingual pretraining for semi-supervised NLP tasks, demonstrating superior performance in low-resource scenarios.

Findings

Outperforms state-of-the-art in low-resource settings

Effective across multiple languages

Highly competitive with supervised baselines

Abstract

Semi-supervised learning through deep generative models and multi-lingual pretraining techniques have orchestrated tremendous success across different areas of NLP. Nonetheless, their development has happened in isolation, while the combination of both could potentially be effective for tackling task-specific labelled data shortage. To bridge this gap, we combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. Compared to strong supervised learning baselines, our semi-supervised classification framework is highly competitive and outperforms the state-of-the-art counterparts in low-resource settings across several languages.