EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training

TL;DR

EVA is a large-scale open-domain Chinese dialogue system leveraging a 2.8 billion parameter pre-trained model trained on the extensive WDC-Dialogue dataset, significantly improving multi-turn conversation quality.

Contribution

This paper introduces EVA, the largest Chinese pre-trained dialogue model with 2.8B parameters, trained on the newly collected 1.4B pair WDC-Dialogue dataset, advancing Chinese dialogue system capabilities.

Findings

EVA outperforms existing Chinese dialogue models in automatic evaluations.

EVA demonstrates superior multi-turn interaction quality in human-bot conversations.

The WDC-Dialogue dataset is the largest Chinese dialogue dataset used for pre-training.

Abstract

Although pre-trained language models have remarkably enhanced the generation ability of dialogue systems, open-domain Chinese dialogue systems are still limited by the dialogue data and the model size compared with English ones. In this paper, we propose EVA, a Chinese dialogue system that contains the largest Chinese pre-trained dialogue model with 2.8B parameters. To build this model, we collect the largest Chinese dialogue dataset named WDC-Dialogue from various public social media. This dataset contains 1.4B context-response pairs and is used as the pre-training corpus of EVA. Extensive experiments on automatic and human evaluation show that EVA outperforms other Chinese pre-trained dialogue models especially in the multi-turn interaction of human-bot conversations.