LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization

TL;DR

LICHEE is a pre-training method that enhances language models by integrating multi-grained tokenization, leading to improved performance on diverse NLU tasks with minimal additional inference cost.

Contribution

The paper introduces LICHEE, a novel pre-training approach that incorporates multi-grained tokenization to improve language model representations across languages.

Findings

Achieves state-of-the-art results on CLUE benchmark.

Improves performance on SuperGLUE tasks.

Effective across Chinese and English NLU tasks.

Abstract

Language model pre-training based on large corpora has achieved tremendous success in terms of constructing enriched contextual representations and has led to significant performance gains on a diverse range of Natural Language Understanding (NLU) tasks. Despite the success, most current pre-trained language models, such as BERT, are trained based on single-grained tokenization, usually with fine-grained characters or sub-words, making it hard for them to learn the precise meaning of coarse-grained words and phrases. In this paper, we propose a simple yet effective pre-training method named LICHEE to efficiently incorporate multi-grained information of input text. Our method can be applied to various pre-trained language models and improve their representation capability. Extensive experiments conducted on CLUE and SuperGLUE demonstrate that our method achieves comprehensive improvements on a wide variety of NLU tasks in both Chinese and English with little extra inference cost incurred, and that our best ensemble model achieves the state-of-the-art performance on CLUE benchmark competition.