Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classification

TL;DR

This paper introduces a retrieval-based in-context learning framework for few-shot hierarchical text classification, leveraging label-aware representations and iterative label management to improve performance on benchmark datasets.

Contribution

It presents the first ICL-based approach for few-shot HTC, incorporating a retrieval database with label-aware representations and a novel divergent contrastive learning objective.

Findings

Achieves state-of-the-art results on benchmark datasets

Outperforms existing methods in few-shot HTC

Demonstrates the effectiveness of label-aware retrieval and iterative label management

Abstract

Hierarchical text classification (HTC) is an important task with broad applications, while few-shot HTC has gained increasing interest recently. While in-context learning (ICL) with large language models (LLMs) has achieved significant success in few-shot learning, it is not as effective for HTC because of the expansive hierarchical label sets and extremely-ambiguous labels. In this work, we introduce the first ICL-based framework with LLM for few-shot HTC. We exploit a retrieval database to identify relevant demonstrations, and an iterative policy to manage multi-layer hierarchical labels. Particularly, we equip the retrieval database with HTC label-aware representations for the input texts, which is achieved by continual training on a pretrained language model with masked language modeling (MLM), layer-wise classification (CLS, specifically for HTC), and a novel divergent contrastive learning (DCL, mainly for adjacent semantically-similar labels) objective. Experimental results on three benchmark datasets demonstrate superior performance of our method, and we can achieve state-of-the-art results in few-shot HTC.