Making Language Model a Hierarchical Classifier

TL;DR

This paper introduces a hierarchical decoder architecture for language models, enabling simultaneous multi-layer decoding to improve performance on hierarchical tasks and reduce computational costs.

Contribution

It proposes a novel hierarchical decoder built by adapting pretrained models, with layers decoding simultaneously, achieving state-of-the-art results on multiple hierarchical NLP tasks.

Findings

Outperforms baselines on WoS, DBpedia, ESconv, and EmpatheticDialogues

Validates meaningful content generation at intermediate layers

Provides theoretical analysis of convergence and efficiency

Abstract

Decoder-only language models, such as GPT and LLaMA, generally decode on the last layer. Motivated by human's hierarchical thinking capability, we propose that a hierarchical decoder architecture could be built with different layers decoding texts simultaneously. Due to limited time and computationally resources, we choose to adapt a pretrained language model into this form of hierarchical decoder. Language heads of the last layer are copied to different selected intermediate layers, and fine-tuned with different task inputs. By thorough experiments, we validate that these selective intermediate layers could be adapted to speak meaningful and reasonable contents, and this paradigm of hierarchical decoder can obtain state-of-the-art performances on multiple tasks such as hierarchical text classification, classification-guided generation, and hierarchical text generation. HdLM outperforms all baselines on WoS, DBpedia, ESconv, EmpatheticDialogues, and several cognitive tests. We also provide thorough theoretical analysis to validate the convergence and computational savings of our methodology. This study suggests the possibility of a generalized hierarchical reasoner, pretraining from scratch.