Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models

TL;DR

This paper introduces a method to incorporate lexical semantics and word structure into Chinese character-based pre-trained models, improving their performance on various NLP tasks by leveraging word-level information.

Contribution

It proposes a novel approach to integrate word semantics into character representations, including a word-to-character alignment and ensemble segmentation to enhance Chinese NLP models.

Findings

Achieves superior performance on multiple Chinese NLP tasks.

Effectively incorporates word semantics into character-based models.

Improves robustness against segmentation errors.

Abstract

Most of the Chinese pre-trained models adopt characters as basic units for downstream tasks. However, these models ignore the information carried by words and thus lead to the loss of some important semantics. In this paper, we propose a new method to exploit word structure and integrate lexical semantics into character representations of pre-trained models. Specifically, we project a word's embedding into its internal characters' embeddings according to the similarity weight. To strengthen the word boundary information, we mix the representations of the internal characters within a word. After that, we apply a word-to-character alignment attention mechanism to emphasize important characters by masking unimportant ones. Moreover, in order to reduce the error propagation caused by word segmentation, we present an ensemble approach to combine segmentation results given by different tokenizers. The experimental results show that our approach achieves superior performance over the basic pre-trained models BERT, BERT-wwm and ERNIE on different Chinese NLP tasks: sentiment classification, sentence pair matching, natural language inference and machine reading comprehension. We make further analysis to prove the effectiveness of each component of our model.