Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis

TL;DR

SentiWSP is a sentiment-aware pre-trained language model that combines word-level and sentence-level pre-training tasks to improve sentiment analysis performance, achieving state-of-the-art results.

Contribution

The paper introduces SentiWSP, a novel PLM with integrated sentiment-aware pre-training at both word and sentence levels, enhancing sentiment understanding.

Findings

Achieves state-of-the-art results on sentiment classification benchmarks.

Effectively captures sentiment information at both word and sentence levels.

Demonstrates significant improvement over existing models.

Abstract

Most existing pre-trained language representation models (PLMs) are sub-optimal in sentiment analysis tasks, as they capture the sentiment information from word-level while under-considering sentence-level information. In this paper, we propose SentiWSP, a novel Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks. The word level pre-training task detects replaced sentiment words, via a generator-discriminator framework, to enhance the PLM's knowledge about sentiment words. The sentence level pre-training task further strengthens the discriminator via a contrastive learning framework, with similar sentences as negative samples, to encode sentiments in a sentence. Extensive experimental results show that SentiWSP achieves new state-of-the-art performance on various sentence-level and aspect-level sentiment classification benchmarks. We have made our code and model publicly available at https://github.com/XMUDM/SentiWSP.