Layer-wise Guided Training for BERT: Learning Incrementally Refined Document Representations

TL;DR

This paper introduces a layer-wise guided training method for BERT that improves document classification performance and parameter utilization by aligning specific layers with hierarchy levels in multilabel text classification.

Contribution

It proposes a novel structured fine-tuning approach for BERT that guides layers to predict labels at specific hierarchy levels, enhancing performance and efficiency.

Findings

Improved classification accuracy on LMTC datasets.

Enhanced parameter utilization in BERT.

Better alignment of model layers with label hierarchy levels.

Abstract

Although BERT is widely used by the NLP community, little is known about its inner workings. Several attempts have been made to shed light on certain aspects of BERT, often with contradicting conclusions. A much raised concern focuses on BERT's over-parameterization and under-utilization issues. To this end, we propose o novel approach to fine-tune BERT in a structured manner. Specifically, we focus on Large Scale Multilabel Text Classification (LMTC) where documents are assigned with one or more labels from a large predefined set of hierarchically organized labels. Our approach guides specific BERT layers to predict labels from specific hierarchy levels. Experimenting with two LMTC datasets we show that this structured fine-tuning approach not only yields better classification results but also leads to better parameter utilization.