Retrieval-augmented Encoders for Extreme Multi-label Text Classification

TL;DR

This paper introduces RAEXMC, a retrieval-augmented encoder framework for extreme multi-label text classification that enhances generalization and memorization without extra trainable parameters, achieving state-of-the-art results and significant speedups.

Contribution

The paper proposes RAEXMC, a retrieval-augmented dual-encoder framework that improves extreme multi-label classification by combining retrieval with contrastive training, eliminating the need for complex model combinations.

Findings

RAEXMC outperforms existing methods on four benchmarks.

RAEXMC achieves over 10x speedup on large-scale datasets.

RAEXMC advances the state-of-the-art DEXML method.

Abstract

Extreme multi-label classification (XMC) seeks to find relevant labels from an extremely large label collection for a given text input. To tackle such a vast label space, current state-of-the-art methods fall into two categories. The one-versus-all (OVA) method uses learnable label embeddings for each label, excelling at memorization (i.e., capturing detailed training signals for accurate head label prediction). In contrast, the dual-encoder (DE) model maps input and label text into a shared embedding space for better generalization (i.e., the capability of predicting tail labels with limited training data), but may fall short at memorization. To achieve generalization and memorization, existing XMC methods often combine DE and OVA models, which involves complex training pipelines. Inspired by the success of retrieval-augmented language models, we propose the Retrieval-augmented Encoders for XMC (RAEXMC), a novel framework that equips a DE model with retrieval-augmented capability for efficient memorization without additional trainable parameter. During training, RAEXMC is optimized by the contrastive loss over a knowledge memory that consists of both input instances and labels. During inference, given a test input, RAEXMC retrieves the top-$K$ keys from the knowledge memory, and aggregates the corresponding values as the prediction scores. We showcase the effectiveness and efficiency of RAEXMC on four public LF-XMC benchmarks. RAEXMC not only advances the state-of-the-art (SOTA) DE method DEXML, but also achieves more than 10x speedup on the largest LF-AmazonTitles-1.3M dataset under the same 8 A100 GPUs training environments.