UER: An Open-Source Toolkit for Pre-training Models

TL;DR

UER is an open-source toolkit that enables flexible assembly and deployment of various pre-training models for NLP, facilitating both reproduction of state-of-the-art models and exploration of new ones.

Contribution

It introduces a modular, assemble-on-demand framework for pre-training NLP models, supporting diverse configurations and enabling rapid development and deployment.

Findings

Built a model zoo with diverse pre-trained models

Achieved new state-of-the-art results on multiple NLP datasets

Demonstrated flexible assembly of pre-training components

Abstract

Existing works, including ELMO and BERT, have revealed the importance of pre-training for NLP tasks. While there does not exist a single pre-training model that works best in all cases, it is of necessity to develop a framework that is able to deploy various pre-training models efficiently. For this purpose, we propose an assemble-on-demand pre-training toolkit, namely Universal Encoder Representations (UER). UER is loosely coupled, and encapsulated with rich modules. By assembling modules on demand, users can either reproduce a state-of-the-art pre-training model or develop a pre-training model that remains unexplored. With UER, we have built a model zoo, which contains pre-trained models based on different corpora, encoders, and targets (objectives). With proper pre-trained models, we could achieve new state-of-the-art results on a range of downstream datasets.