SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification

TL;DR

This paper introduces SPIRIT, a new method for hierarchical text classification that uses structural entropy to improve model performance.

Contribution

SPIRIT introduces a novel prefix tuning approach guided by structural entropy for hierarchical text classification.

Findings

SPIRIT outperforms existing methods on four popular hierarchical text classification datasets.

The method uses structural entropy to capture the essential hierarchy of labels.

A depth-wise reparameterization strategy enhances optimization and model performance.

Abstract

Hierarchical text classification (HTC) is a challenging task that requires classifiers to solve a series of multi-label subtasks considering hierarchical dependencies among labels. Recent studies have introduced prompt tuning to create closer connections between the language model (LM) and the complex label hierarchy. However, we find that the model’s attention to the prompt gradually decreases as the prompt moves from the input to the output layer, revealing the limitations of previous prompt tuning methods for HTC. Given the success of prefix tuning-based studies in natural language understanding tasks, we introduce Structural entroPy guIded pRefIx Tuning (SPIRIT). Specifically, we extract the essential structure of the label hierarchy via structural entropy minimization and decode the abstractive structural information as the prefix to prompt all intermediate layers in the LM. Additionally, a depth-wise reparameterization strategy is developed to enhance optimization and propagate the prefix throughout the LM layers. Extensive evaluation on four popular datasets demonstrates that SPIRIT achieves a state-of-the-art performance.

Figures

Full text

1. Introduction

Hierarchical text classification (HTC) poses a significant challenge, as the classifier must simultaneously comprehend the semantics in the text and the hierarchical correlations inherent in the label set. A popular approach for HTC is to express the label set as a label hierarchy (intrinsically, a directed acyclic graph) and adopt a graph neural network to extract structural information using a dual-encoder framework in conjunction with the LM [1]. Furthermore, to enhance the collaboration between the text and the structure encoders, various deep learning techniques, such as semantic matching [2], information maximization [3], and contrastive learning [4,5], are introduced to HTC. To fully exploit and reuse the capabilities of pretrained language models, recent studies have proposed prompt tuning methods [6,7,8], achieving great improvements in HTC.

However, two observations reveal the limitations of vanilla prompt tuning and spark our interest in prefix tuning for HTC: (1) As depicted in Figure 1, HPT [6], a prominent prompt tuning method for HTC, where the attention scores on the prompt contents potentially display continual downward trends as the input dataflows toward the output layer. (2) Other prior research indicates that prompt tuning encounters performance bottlenecks on natural language understanding tasks, whereas the involvement of prefix tuning [9,10] yields performance comparable to fine-tuning [11]. Considering the two observations, prefix tuning may be a better choice than prompt tuning for HTC, as the prefix could persistently recall the label hierarchy to the language model.

Another issue preventing the prompt learning method from breakthroughs in HTC is that natural language-based prompts fail to convey the intricate label hierarchy [6], even when using advanced language models like ChatGPT [5]. In contrast, we consider generating prompts from a structural perspective. In light of structural entropy [12], which measures the uncertainty of a graph and provides the essential structure of the graph formulated in coding trees, we aim to construct structural prefixes for HTC with the principle of structural entropy minimization. Specifically, we implement an algorithm to realize optimal coding tree construction, uncovering the underlying structure of label hierarchies. Subsequently, we design a structural prefix network to decode the structural information embedded in the coding trees and generate initial prefix representations. Additionally, a depth-wise reparameterization is originated to propagate the prompt tokens to each hidden layer, allowing prefixes to be better adapted to the LM and reduce the training burden. In comparison with other supervised-only, contrastive learning, and prompt tuning methods, SPIRIT achieves superior performance on four common datasets. Overall, the contributions of our work can be summarized as follows:

• We propose a structural entropy guided prefix tuning method named SPIRIT for hierarchical text classification. To the best of our knowledge, this paper is the first introduction of prefix tuning to HTC.
• To construct hierarchy-aware prefixes, we decode the essential structure of label hierarchies with a structural entropy minimization algorithm, providing abstractive information to be encoded into the prefix.
• Given the essential structure of label hierarchies, a structural prefix network is proposed to construct initial prefix embeddings for prompting all hidden layers in the LM. Further, a depth-wise reparameterization strategy is originated for adapting structural prefixes to the LM.
• Experiments on four datasets demonstrate the effectiveness of SPIRIT For reproducibility, source code is available at https://github.com/Rooooyy/SPIRIT (accessed on 11 January 2025).

2. Preliminary

Hierarchical Text Classification. In a specific hierarchical text classification task, there exists L classification subtasks $[eqn]$ , where $[eqn]$ is a multi-label classification task at the l-th level. The total number of label is denoted as $[eqn]$ . Furthermore, dependencies among labels in adjacent levels, derived from their semantics, can be formulated as a directed graph. This graph—known as the label hierarchy—contains edges only from nodes at the $[eqn]$ -th level to those at the l-th level (i.e., no self-loops or cross-level linking), capturing the hierarchical dependencies embedded in the label set. The ground truth labels always co-occur as paths from the root node to the leaf nodes. Given a piece of text $[eqn]$ , the classifier should predict the labels in $[eqn]$ concerning the hierarchy.

Structural Entropy. Structural entropy [12] extends Shannon entropy on graph structures, measuring the uncertainty of a graph. Coding trees are carriers for structural entropy, revealing the essential structure of a graph. Formally, the structural entropy of a graph $[eqn]$ is defined as follows:

$$[eqn]$$

where $[eqn]$ is a non-root node of coding tree $[eqn]$ which represents a subset of $[eqn]$ , $[eqn]$ is the parent node of $[eqn]$ on the coding tree. $[eqn]$ represents the out degree of $[eqn]$ on G. $[eqn]$ denotes the volume of graph G while $[eqn]$ and $[eqn]$ are the total degrees of nodes in $[eqn]$ and $[eqn]$ , respectively. The height of the coding tree should be fixed to formulate a certain coding scheme. Accordingly, the K-dimensional structural entropy of the graph G is the minimal structural entropy determined by coding tree T with height no greater than K, formally,

$$[eqn]$$

Coding Tree. The coding tree $[eqn]$ of a graph $[eqn]$ is a tree with the following properties: (1) Every node $[eqn]$ is assigned with a non-empty subset of V, denoted as $[eqn]$ . The root node $[eqn]$ of $[eqn]$ marks all nodes in G, i.e., $[eqn]$ . For each leaf node $[eqn]$ , $[eqn]$ contains a single node $[eqn]$ . (2) For each non-leaf node $[eqn]$ , its i-th child node is denoted as $[eqn]$ , $[eqn]$ is the union of its child nodes, i.e., $[eqn]$ (there is a proposition that nodes on the same level as $[eqn]$ provide a partition of V). Additionally, a notation list is given in Appendix A.1.

3. Methodology

In this section, we elaborate on the design of SPIRIT in Figure 2.

3.1. Input Prompt

Following [6], we construct dynamic soft prompts incorporating hierarchy-aware information. The quantity of virtual tokens is equal to the depth of the label hierarchy. Each token corresponds to a layer in the hierarchy with the same index. Furthermore, each layer’s prompt token is appended by a special token for layer-wise classification. Formally, given a document $[eqn]$ , the input to the LM is as follows:

$$[eqn]$$

where $[eqn]$ and $[eqn]$ denote the embeddings of the <CLS> and <SEP> tokens, respectively. $[eqn]$ is initialized with <MASK> token for layer-wise multi-label prediction. $[eqn]$ are prompt tokens produced by a prompt encoder. Specifically, L virtual nodes are added to the label hierarchy and each virtual node is linked with all label nodes in the same layer. For representation learning, the virtual nodes are randomly initialized while the label nodes are initialized with their textual description encoded by the LM. Thereafter, a one-layer graph attention network (GAT) [13] is adopted to model label correlations on the augmented label hierarchy graph. The embeddings of virtual nodes fetched by GAT are taken as continuous prompts, where the information of the label hierarchy is condensed. Given a node u on the label hierarchy graph within virtual nodes, the information propagation process of GAT is formulated as

$$[eqn]$$

where $[eqn]$ denotes the neighbors for node u, $[eqn]$ is a normalization constant, and $[eqn]$ is the trainable parameter. Then, we filter out those embeddings of the virtual nodes $[eqn]$ to construct the prompt.

3.2. Structural Entropy Guided Prefix

Simply prompting the model at input layer might not be sufficient for HTC as the label hierarchy provides rules for classification, rather than merely serving as task instructions. To continually reinforce the LM’s memory of the label hierarchy, we replay the knowledge of labels via prefix tuning [9]. This involves introducing prompt tokens for all intermediate layers in the LM. Intuitively, the intermediate layers should be more attentive to coarse-grained information compared with the input layer. Further, layers at higher levels should receive and produce denser information than those at lower levels. Based on these assumptions, we utilize the essential structural information provided by coding trees, which preserve minimal structural entropy [12], to construct prefix tokens. We also propose a depth-wise reparameterization strategy for broadcasting the prefixes to LM layers at different depths.

Coding Tree Construction. Structural entropy could measure the uncertainty of a graph. Minimizing structural entropy, a coding tree can be constructed to reveal the essential structure of the graph. Converting the label hierarchy into such a coding tree would make the intermediate LM layers easier to understand. To this end, we design an efficient algorithm to construct the optimal coding tree obtaining minimal K dimensional structural entropy. Given a graph $[eqn]$ , let $[eqn]$ be a partition of V, where $[eqn]$ is called a community. The algorithm is implemented based on three atom functions as follows:

Definition 1. Function $[eqn]$ . Given any two communities $[eqn]$ , merge function $[eqn]$ merges $[eqn]$ and $[eqn]$ into a new community $[eqn]$ , i.e., $[eqn]$ .

Here, we denote the reduction in structural entropy after calling $[eqn]$ as $[eqn]$ . According to Equation (1), we have

$$[eqn]$$

Definition 2. Function $[eqn]$ . Given a graph G and a partition $[eqn]$ , function $[eqn]$ squeezes G in the following procedure. For all communities $[eqn]$ in $[eqn]$ , nodes marked by $[eqn]$ on graph G are squeezed to a new node $[eqn]$ , and the edge weights of $[eqn]$ to others is the summation of those original nodes in  $[eqn]$ .

Definition 3. Function $[eqn]$ . Given a coding tree $[eqn]$ and a partition $[eqn]$ . $[eqn]$ will update coding tree $[eqn]$ by taking all communities in $[eqn]$ as leaf nodes of $[eqn]$ , leading to an increase in height.

Initially, we adopt each node in the graph as a single community, and then iteratively execute $[eqn]$ and $[eqn]$ until a K-dimensional coding tree is constructed. Given a sub-graph $[eqn]$ of G in the k-th iteration, we merge the communities with the maximal $[eqn]$ greedily until there are no communities satisfying $[eqn]$ , thereby achieving the minimal K-dimensional structural entropy. The complete procedure is shown in Algorithm 1. Furthermore, a case study is provided in Appendix A.2 for a better illustration of Algorithm 1. **Algorithm 1: **Coding Tree Construction

Theorem 1. Given a graph $[eqn]$ , for any target height K, Algorithm 1 is in the complexity of $[eqn]$ .

Proof. Lines 1–6 of Algorithm 1 merge the nodes of $[eqn]$ from bottom to top. For each iteration in $[eqn]$ , $[eqn]$ generates $[eqn]$ new nodes for $[eqn]$ . Then, $[eqn]$ traverses and squeezes these nodes; this results in the retention of nodes no more than $[eqn]$ . Therefore, the node size of $[eqn]$ is lower than that of a binary tree $[eqn]$ constructed on the $[eqn]$ leaf nodes. The complexity of traversing $[eqn]$ is calculated as follows

$$[eqn]$$

Algorithm 1 executes K linear traversals and operations on $[eqn]$ . The total number of operations is twice the node size of $[eqn]$ . Thus, the complexity of Algorithm 1 is also $[eqn]$ . □

Structural Prefix Network. After calling Algorithm 1 on the label hierarchy, we obtain a coding tree $[eqn]$ with height K. For representation learning, we initialize the leaf node embeddings $[eqn]$ with the textual descriptions of their corresponding labels. Here, non-leaf nodes embeddings $[eqn]$ still remain unknown. To fetch intermediate node embeddings, we design a message-passing network as follows,

$$[eqn]$$

where $[eqn]$ are learnable weights in the k-th layer, and $[eqn]$ denotes the child nodes of $[eqn]$ .

Depth-wise Reparametrization. In light of the introduction of reparametrization in previous prefix tuning studies [9,11], we propose a depth-wise reparametrization to construct prefix embeddings. First, we take the average of node embeddings at each layer in $[eqn]$ as the initial prefix Q, formally,

$$[eqn]$$

where $[eqn]$ denotes a group of learnable parameters. Next, given B LM layers, the final prefix $[eqn]$ , where $[eqn]$ denotes the sigmoid function. $[eqn]$ is the relative distance of k and b, formally,

$$[eqn]$$

where $[eqn]$ for avoiding calculation error. The intuition of Equation (9) is that the node embeddings at lower levels of coding tree $[eqn]$ would be more suitable for prompting layers next to the input layer in the LM while node embeddings at higher levels, preserving denser information, might be better prefix for layers beneath the output layer.

3.3. Model Training and Prediction

Hierarchy-constrained Prediction. At the output layer, the LM could predict the logits $[eqn]$ that $[eqn]$ corresponds to each label, formally,

$$[eqn]$$

where $[eqn]$ is a single logit. When $[eqn]$ , the model will predict the corresponding label $[eqn]$ as a result. At the inference stage, when the l-th layer prompt is observed, we further restrict the model to predict only the labels from $[eqn]$ .

ZLPR Loss. A majority of previous methods [1,2,4,14] adopt binary cross entropy (BCE) loss as the optimization target, which intrinsically degrades HTC into a naive multi-label classification. Following recent works [5,6] in HTC, we alternatively utilize the “Zero-bounded Log-sum-exp & Pairwise Rank-based” (ZLPR) loss proposed by [15], formally,

$$[eqn]$$

where $[eqn]$ denote the target labels and non-target labels in $[eqn]$ , dynamically obtained using the ground truths. ZLPR loss explicitly captures the label correlations in the label hierarchy, which is proved beneficial for HTC [5,6].

Masked LM Training. We keep the original masking strategy and loss as BERT [16] to train our model, where 15% words of the text are randomly masked. The final loss $[eqn]$ is the summation of ZLPR loss with regard to classification subtasks $[eqn]$ at different levels and the MLM loss $[eqn]$ , formally,

$$[eqn]$$

4. Experimental Evaluation

Datasets and Evaluation Metrics. Experiments are conducted on four popular datasets in HTC: WOS [17], RCV1-v2 [18], NYTimes [19], and BGC (https://www.inf.uni-hamburg.de/en/inst/ab/lt/resources/data/blurb-genre-collection.html, accessed on 7 May 2024). We randomly split WOS and RCV1-v2 following [6] while keeping the official splits for NYTimes and BGC. The statistics of these datasets are shown in Table 1. Model performance is measured by Micro-F1 and Macro-F1 [20]. Micro-F1 is the harmonic mean of the overall precision and recall of all the test instances, while Macro-F1 is the average F1 score of each category. Thus, Micro-F1 reflects the performance on more frequent labels, while Macro-F1 treats labels equally.

Implementation Details. We adopt BERT [16] as the LM to be prompted and update its parameters with Adam [21] as the initial learning rate is $[eqn]$ . For simplicity, the hidden size d of all the learnable weights is set to 768 if not mentioned. The model is trained on a single Nvidia V100 32GB GPU with batch size of 24. Training will be terminated if the Macro-F1 does not increase on the dev set for five epochs. The optimal height K of coding trees are $[eqn]$ for WOS, RCV1-v2, NYTimes, and BGC, respectively. For all text samples, we truncate them to a maximum length of 500.

Baselines. All baselines adopt BERT [16]. The performance gaps compared with the vanilla BERT show the significance of their original parts. HiAGM [1], HTCInfoMax [3], HiMatch [2], and HiTIN [14] are all supervised models based on the dual-encoder framework within various techniques to enhance the combination. HGCLR [4] is the pioneer investigation of contrastive learning for HTC while HJCL [5] proposed an instance-wise and label-wise joint supervised contrastive loss. HPT [6] is the inaugural work to introduce prompt tuning for HTC, where a dynamic virtual template and label words are proposed to bridge the gap between MLM and HTC. HBGL [7] pretrains a set of label embeddings with a pretext task and feeds those of ground truth labels to LM during training, which could be treated as a pretrained prompt tuning [22] manner.

4.1. Results and Analysis

The main results are shown in Table 2. We can observe that SPIRIT brings 2.26 and 4.69 average improvements to vanilla BERT on Micro-F1 and Macro-F1, respectively. Compared to all the baselines, SPIRIT obtains a state-of-the-art performance on seven out of eight of the metrics. The Macro-F1 performance of SPIRIT is second only to that of HJCL; however, it is notable that HJCL introduces both ZLPR loss and the supervised contrastive learning loss, intrinsically utilizing more supervision signals. In contrast, MLM loss is naturally at a disadvantage due to the self-supervised feature.

Among those prompt tuning methods, SPIRIT outperforms all others across all datasets. Our model significantly outperforms when using datasets with more complex label hierarchies (i.e., RCV1-v2 and NYTimes), while the gaps between it and the runner-up are smaller on WOS. A probable reason is that WOS has only two levels of hierarchy and the ground truths are all single-path, which could not fully present the advantages of structural prefix tuning. Moreover, the ups and downs of HBGL’s ranking on the three datasets prove that its pretrained label embeddings are derived more from semantic information than structural information.

4.2. Ablation Studies

Prefix Tuning Only Works in the Framework of SPIRIT. To track the source of performance improvements, we conducted ablation studies on both WOS and NYTimes in Table 3. Among the four datasets, WOS has the shallowest label hierarchy (2), while that of NYTimes is the deepest (8). We first replace the structural prefix network (Section 3.2) with a two-layer MLP: $[eqn]$ to generate the final prefix, where $[eqn]$ are layerwise prompts (Section 3.1). $[eqn]$ and $[eqn]$ . This is a common prefix tuning network adopted in [9,11]. The performance gaps demonstrate the superiority of structural entropy guided prefixes. Next, we skip Equation (9) and directly feed B copies of Q to the LM (r.p. $[eqn]$ ). The degraded results show the necessity of simulating information densification. Further, we replace $[eqn]$ with a random initialized matrix $[eqn]$ , providing a lower bound as random prefix tuning. The above results jointly suggest the integrity of SPIRIT.

Effects of the K-dimensional Structural Entropy. The only hyperparameter of SPIRIT to be tuned is K, which represents the dimension of structural entropy and the height of coding trees. To investigate the effects of K, we train SPIRIT with the value of K ranges in $[eqn]$ while keeping other settings the same. Figure 3 shows the test performance of different height coding trees on WOS and NYTimes. As K grows, the performance of SPIRIT shows a trend of degradation. Higher coding trees may involve more explosive gradients, resulting in unstable optimization.

4.3. Evaluation on Path Consistency

We hold the same view as [8,23] that path consistency is crucial to HTC. We further evaluate models with P metrics [8] and C metrics [23], both of which are stricter than the vanilla Micro-F1 and Macro-F1 metrics. In P metrics, only if all labels in one path are predicted accurately is the prediction is considered correct in the confusion matrix. C metrics emphasize that a node label is considered predicted correctly only when all its ancestor nodes are correct. As depicted in Figure 4, our model can better convey the labeling hierarchy at different depths and outperforms HPT in P metrics and C metrics across the board. In comparison with the ablation model “r.p. $[eqn]$ ”, denoted as “SPIRIT (ones)” in Figure 4, SPIRIT also exhibits superior performance, highlighting the necessity of depth-wise reparameterization (Section 3.2).

4.4. Comparison with Large Language Models

Table 4 presents the performance of two emerging large language models, GPT-3.5-Turbo [24] and GPT-4o-0513 [25]. Following [5], we adopt the same prompt template to evaluate the in context learning ability of GPT-4o API (https://openai.com/api/). We also repeat the experiments utilizing a new prompt template presented in Table 5. It can be found that the instruction-tuned large language models are still inferior to the supervised fine-tuned models. Further, we could tell that the reasoning capability of large language models may not generalize to structural information, and the introduction of a structural encoder is necessary.

4.5. Qualitative Analysis

We illustrate a case study in Figure 5 and Table 6. The test case is from NYTimes [19]. Benefiting from the structural prefix tuning and depth-wise reparameterization, our model exhibits a more profound understanding of the hierarchy, successfully predicting all three paths of ground truths. In contrast, due to providing prompts in the input layer only, HPT rapidly forgets the label hierarchy, resulting in only one predicted path. The attention scores in SPIRIT (ones), surpass SPIRIT at the 3→9-th layer but drop dramatically at layer 10→11. Equally weighted prompts confuse the model output, resulting in only two successfully predicted paths.

To dynamically demonstrate the effects of structural prefixes and depth-wise reparameterization, we plot the attention scores of text tokens with respect to the structural as heat maps in Figure 6. Structural prefixes are given more attention by the model at both the input and output layers, which demonstrates structural prefix tuning plays an important role in input and output. In the middle layer of the model, influenced by depth-wise reparameterization, the model ignores the zeroth and second prefixes and pays more attention to the intermediate prefixes, which is in line with our expectations.

5. Related Work

Hierarchical Text Classification. Existing works for HTC could be categorized into local and global approaches [1]. Local approaches build multiple models for labels in different levels in the hierarchy, conveying the information from models in the upper levels to those in the bottom [17,26,27,28]. On the contrary, global studies treat HTC as a flat multi-label classification problem [1,2,3,20,29,30,31,32]. With the advent of pretrained language models (PLMs), numerous studies attempt to solve HTC tasks with PLMs. Techniques derived from pretrained language modeling such as contrastive learning [4,5] and prompt tuning [6,7] are successfully applied to HTC. The boundary between local and global methods is blurring as lots of efforts are paid to utilize the mixture of local and global information [5,7].

Prompt Tuning. There are several widely studied prompt tuning manners: (1) Naive prompt tuning, where continuous prompts are directly fed into the input layer for jointly modeling with the original inputs [10,33,34,35]. (2) Prefix tuning, where continuous prompts are not only added to the input but also to the hidden states [9,11]. (3) P-tuning, where a prompt encoder is involved to capture the correlation among continuous prompts [11,36]. (4) Pretrained prompt tuning, where continuous prompts are pretrained together with the language model [22]. Prompt tuning methods for HTC tend to produce continuous prompts based on the label hierarchy. According to the above taxonomy, HPT [6] could be exactly recognized as a p-tuning method since it generates virtual prompt tokens with a graph attention network [13], which acts as the prompt encoder. HBGL [7] could be referred to as a pretrained prompt tuning method where the label embeddings are pretrained under a pretext task along with BERT [16].

Structural Entropy. Structural entropy [12] is a natural extension of Shannon entropy [37] in a structural system, capable of measuring the system’s structural complexity. Non-Euclidean data, especially graph data, create a typical structured system. The structural entropy of a graph is defined as the average length of the codewords obtained by a random walk under a specific coding scheme, namely coding tree [12]. In the past few years, structural entropy has been successfully applied in various domains [38,39,40,41,42,43,44,45].

6. Conclusions

We introduce SPIRIT, a prefix tuning framework consisting of a structural entropy minimization algorithm, a structural prefix network, and a depth-wise reparameterization strategy. For HTC, structural entropy-guided prefixes could significantly improve the memory of language models for label hierarchy. Depth-wise reparameterization effectively adapts the prefixes to LM layers at different depths. Moreover, random prefixes and repetitive prefixes demonstrate that structural entropy-guided prefix tuning is feasible for improving HTC performance.