Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese

TL;DR

Mengzi is a family of lightweight, efficient Chinese pre-trained models that achieve state-of-the-art results across various NLP and vision tasks without increasing model size or complexity.

Contribution

We introduce Mengzi, a set of lightweight, powerful Chinese pre-trained models that outperform existing models on benchmark tasks using optimized training and fine-tuning techniques.

Findings

Achieved new state-of-the-art results on CLUE benchmark

Models are simpler yet more powerful than existing Chinese PLMs

No architecture modifications needed for deployment

Abstract

Although pre-trained models (PLMs) have achieved remarkable improvements in a wide range of NLP tasks, they are expensive in terms of time and resources. This calls for the study of training more efficient models with less computation but still ensures impressive performance. Instead of pursuing a larger scale, we are committed to developing lightweight yet more powerful models trained with equal or less computation and friendly to rapid deployment. This technical report releases our pre-trained model called Mengzi, which stands for a family of discriminative, generative, domain-specific, and multimodal pre-trained model variants, capable of a wide range of language and vision tasks. Compared with public Chinese PLMs, Mengzi is simple but more powerful. Our lightweight model has achieved new state-of-the-art results on the widely-used CLUE benchmark with our optimized pre-training and fine-tuning techniques. Without modifying the model architecture, our model can be easily employed as an alternative to existing PLMs. Our sources are available at https://github.com/Langboat/Mengzi.