An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels

TL;DR

This paper empirically evaluates various large-scale multi-label text classification methods, demonstrating the superiority of hierarchical and Transformer-based models, and introduces new approaches for improved few and zero-shot learning leveraging label hierarchies.

Contribution

It provides the first comprehensive empirical comparison of LMTC methods, introduces a new state-of-the-art combining BERT with LWANs, and proposes models leveraging label hierarchies for better few and zero-shot learning.

Findings

Hierarchical Probabilistic Label Trees outperform flat LWANs.

Transformer-based models outperform previous state-of-the-art in two datasets.

New models leveraging label hierarchies improve few and zero-shot learning.

Abstract

Large-scale Multi-label Text Classification (LMTC) has a wide range of Natural Language Processing (NLP) applications and presents interesting challenges. First, not all labels are well represented in the training set, due to the very large label set and the skewed label distributions of LMTC datasets. Also, label hierarchies and differences in human labelling guidelines may affect graph-aware annotation proximity. Finally, the label hierarchies are periodically updated, requiring LMTC models capable of zero-shot generalization. Current state-of-the-art LMTC models employ Label-Wise Attention Networks (LWANs), which (1) typically treat LMTC as flat multi-label classification; (2) may use the label hierarchy to improve zero-shot learning, although this practice is vastly understudied; and (3) have not been combined with pre-trained Transformers (e.g. BERT), which have led to state-of-the-art results in several NLP benchmarks. Here, for the first time, we empirically evaluate a battery of LMTC methods from vanilla LWANs to hierarchical classification approaches and transfer learning, on frequent, few, and zero-shot learning on three datasets from different domains. We show that hierarchical methods based on Probabilistic Label Trees (PLTs) outperform LWANs. Furthermore, we show that Transformer-based approaches outperform the state-of-the-art in two of the datasets, and we propose a new state-of-the-art method which combines BERT with LWANs. Finally, we propose new models that leverage the label hierarchy to improve few and zero-shot learning, considering on each dataset a graph-aware annotation proximity measure that we introduce.