EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing

TL;DR

EasyNLP is a user-friendly toolkit that simplifies building, training, and deploying NLP models, supporting advanced features like knowledge-enhanced pre-training and few-shot learning, with successful industrial deployment.

Contribution

It introduces a comprehensive, easy-to-use NLP toolkit that integrates multiple algorithms and deployment capabilities, facilitating industrial applications of large-scale pre-trained models.

Findings

Powered over ten Alibaba business units

Seamless integration with Alibaba Cloud PAI

Open-source availability on GitHub

Abstract

The success of Pre-Trained Models (PTMs) has reshaped the development of Natural Language Processing (NLP). Yet, it is not easy to obtain high-performing models and deploy them online for industrial practitioners. To bridge this gap, EasyNLP is designed to make it easy to build NLP applications, which supports a comprehensive suite of NLP algorithms. It further features knowledge-enhanced pre-training, knowledge distillation and few-shot learning functionalities for large-scale PTMs, and provides a unified framework of model training, inference and deployment for real-world applications. Currently, EasyNLP has powered over ten business units within Alibaba Group and is seamlessly integrated to the Platform of AI (PAI) products on Alibaba Cloud. The source code of our EasyNLP toolkit is released at GitHub (https://github.com/alibaba/EasyNLP).