HiLight: A Hierarchy-aware Light Global Model with Hierarchical Local ConTrastive Learning

TL;DR

This paper introduces HiLight, a lightweight hierarchy-aware model for hierarchical text classification that employs hierarchical local contrastive learning to efficiently incorporate hierarchical information without complex structure encoders.

Contribution

The paper proposes HiLight, a novel lightweight model with hierarchical local contrastive learning, addressing scale issues of structure encoders in HTC.

Findings

HiLight achieves competitive performance on benchmark datasets.

Hierarchical local contrastive learning effectively encodes hierarchical information.

The model is more efficient and scalable than existing HTC models.

Abstract

Hierarchical text classification (HTC) is a special sub-task of multi-label classification (MLC) whose taxonomy is constructed as a tree and each sample is assigned with at least one path in the tree. Latest HTC models contain three modules: a text encoder, a structure encoder and a multi-label classification head. Specially, the structure encoder is designed to encode the hierarchy of taxonomy. However, the structure encoder has scale problem. As the taxonomy size increases, the learnable parameters of recent HTC works grow rapidly. Recursive regularization is another widely-used method to introduce hierarchical information but it has collapse problem and generally relaxed by assigning with a small weight (ie. 1e-6). In this paper, we propose a Hierarchy-aware Light Global model with Hierarchical local conTrastive learning (HiLight), a lightweight and efficient global model only consisting of a text encoder and a multi-label classification head. We propose a new learning task to introduce the hierarchical information, called Hierarchical Local Contrastive Learning (HiLCL). Extensive experiments are conducted on two benchmark datasets to demonstrate the effectiveness of our model.