Span Fine-tuning for Pre-trained Language Models

TL;DR

This paper introduces a flexible span fine-tuning method for pre-trained language models that adaptively incorporates span-level information during downstream task training, improving performance and efficiency.

Contribution

It proposes a novel span fine-tuning approach that dynamically determines span settings during task-specific training, unlike fixed span methods.

Findings

Significantly improves PrLM performance on GLUE benchmark

Offers flexible span setting during fine-tuning

Enhances efficiency over previous span-level methods

Abstract

Pre-trained language models (PrLM) have to carefully manage input units when training on a very large text with a vocabulary consisting of millions of words. Previous works have shown that incorporating span-level information over consecutive words in pre-training could further improve the performance of PrLMs. However, given that span-level clues are introduced and fixed in pre-training, previous methods are time-consuming and lack of flexibility. To alleviate the inconvenience, this paper presents a novel span fine-tuning method for PrLMs, which facilitates the span setting to be adaptively determined by specific downstream tasks during the fine-tuning phase. In detail, any sentences processed by the PrLM will be segmented into multiple spans according to a pre-sampled dictionary. Then the segmentation information will be sent through a hierarchical CNN module together with the representation outputs of the PrLM and ultimately generate a span-enhanced representation. Experiments on GLUE benchmark show that the proposed span fine-tuning method significantly enhances the PrLM, and at the same time, offer more flexibility in an efficient way.