MOSLD-Bench: Multilingual Open-Set Learning and Discovery Benchmark for Text Categorization

TL;DR

This paper introduces MOSLD-Bench, a comprehensive multilingual benchmark for open-set learning and discovery in text categorization, along with a novel framework for discovering and learning new classes across multiple languages.

Contribution

It presents the first multilingual open-set learning and discovery benchmark for text classification and proposes a new multi-stage framework for continuous class discovery and learning.

Findings

Benchmark includes 960K samples across 12 languages.

Evaluated multiple language models on the benchmark.

Framework demonstrates effective discovery and learning of new classes.

Abstract

Open-set learning and discovery (OSLD) is a challenging machine learning task in which samples from new (unknown) classes can appear at test time. It can be seen as a generalization of zero-shot learning, where the new classes are not known a priori, hence involving the active discovery of new classes. While zero-shot learning has been extensively studied in text classification, especially with the emergence of pre-trained language models, open-set learning and discovery is a comparatively new setup for the text domain. To this end, we introduce the first multilingual open-set learning and discovery (MOSLD) benchmark for text categorization by topic, comprising 960K data samples across 12 languages. To construct the benchmark, we (i) rearrange existing datasets and (ii) collect new data samples from the news domain. Moreover, we propose a novel framework for the OSLD task, which integrates multiple stages to continuously discover and learn new classes. We evaluate several language models, including our own, to obtain results that can be used as reference for future work. We release our benchmark at https://github.com/Adriana19Valentina/MOSLD-Bench.