ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders

TL;DR

This paper introduces ZEN 2.0, an improved pre-trained text encoder that incorporates n-gram information, enhancing performance across multiple languages and domains in various NLP tasks.

Contribution

The paper presents a novel approach to pre-train n-gram-enhanced encoders using large-scale data and advanced techniques, demonstrating broad applicability and state-of-the-art results.

Findings

Achieved new state-of-the-art performance on multiple NLP benchmarks.

Proved the encoder's effectiveness across different languages and domains.

Enhanced understanding of semantic evidence through n-gram integration.

Abstract

Pre-trained text encoders have drawn sustaining attention in natural language processing (NLP) and shown their capability in obtaining promising results in different tasks. Recent studies illustrated that external self-supervised signals (or knowledge extracted by unsupervised learning, such as n-grams) are beneficial to provide useful semantic evidence for understanding languages such as Chinese, so as to improve the performance on various downstream tasks accordingly. To further enhance the encoders, in this paper, we propose to pre-train n-gram-enhanced encoders with a large volume of data and advanced techniques for training. Moreover, we try to extend the encoder to different languages as well as different domains, where it is confirmed that the same architecture is applicable to these varying circumstances and new state-of-the-art performance is observed from a long list of NLP tasks across languages and domains.