CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations

TL;DR

CLOWER is a pre-trained language model for Chinese that uses contrastive learning to effectively integrate word and character representations, improving semantic understanding and downstream task performance.

Contribution

It introduces a novel contrastive learning approach over word and character representations, enhancing semantic interaction in Chinese PLMs without altering existing pipelines.

Findings

Outperforms several state-of-the-art baselines on downstream tasks.

Effectively encodes semantic relations between words and characters.

Easily integrated into existing PLMs without pipeline modifications.

Abstract

Pre-trained Language Models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. Various Chinese PLMs have been successively proposed for learning better Chinese language representation. However, most current models use Chinese characters as inputs and are not able to encode semantic information contained in Chinese words. While recent pre-trained models incorporate both words and characters simultaneously, they usually suffer from deficient semantic interactions and fail to capture the semantic relation between words and characters. To address the above issues, we propose a simple yet effective PLM CLOWER, which adopts the Contrastive Learning Over Word and charactER representations. In particular, CLOWER implicitly encodes the coarse-grained information (i.e., words) into the fine-grained representations (i.e., characters) through contrastive learning on multi-grained information. CLOWER is of great value in realistic scenarios since it can be easily incorporated into any existing fine-grained based PLMs without modifying the production pipelines.Extensive experiments conducted on a range of downstream tasks demonstrate the superior performance of CLOWER over several state-of-the-art baselines.