Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text Classification

TL;DR

This paper introduces MICoL, a novel contrastive learning approach that leverages document metadata to improve zero-shot multi-label text classification, especially for infrequent labels, without requiring annotated training data.

Contribution

MICoL is the first to exploit document metadata for contrastive learning in zero-shot LMTC, significantly improving performance and addressing long-tail label issues.

Findings

MICoL outperforms existing zero-shot baselines.

MICoL matches supervised methods trained on large labeled datasets.

MICoL predicts more infrequent labels, reducing long-tail performance issues.

Abstract

Large-scale multi-label text classification (LMTC) aims to associate a document with its relevant labels from a large candidate set. Most existing LMTC approaches rely on massive human-annotated training data, which are often costly to obtain and suffer from a long-tailed label distribution (i.e., many labels occur only a few times in the training set). In this paper, we study LMTC under the zero-shot setting, which does not require any annotated documents with labels and only relies on label surface names and descriptions. To train a classifier that calculates the similarity score between a document and a label, we propose a novel metadata-induced contrastive learning (MICoL) method. Different from previous text-based contrastive learning techniques, MICoL exploits document metadata (e.g., authors, venues, and references of research papers), which are widely available on the Web, to derive similar document-document pairs. Experimental results on two large-scale datasets show that: (1) MICoL significantly outperforms strong zero-shot text classification and contrastive learning baselines; (2) MICoL is on par with the state-of-the-art supervised metadata-aware LMTC method trained on 10K-200K labeled documents; and (3) MICoL tends to predict more infrequent labels than supervised methods, thus alleviates the deteriorated performance on long-tailed labels.