Representing Affect Information in Word Embeddings

TL;DR

This paper investigates whether affective meanings like valence, arousal, and dominance are encoded in large neural network word embeddings, finding that affect information is only captured when models are fine-tuned on emotion-related tasks.

Contribution

It provides an analysis of how affect information is represented in different types of word embeddings, highlighting the conditions under which affect is encoded.

Findings

Vanilla BERT embeddings do not encode affect information.

Fine-tuning on emotion tasks enhances affect encoding.

Contextualized embeddings with emotion-rich data better capture affect.

Abstract

A growing body of research in natural language processing (NLP) and natural language understanding (NLU) is investigating human-like knowledge learned or encoded in the word embeddings from large language models. This is a step towards understanding what knowledge language models capture that resembles human understanding of language and communication. Here, we investigated whether and how the affect meaning of a word (i.e., valence, arousal, dominance) is encoded in word embeddings pre-trained in large neural networks. We used the human-labeled dataset as the ground truth and performed various correlational and classification tests on four types of word embeddings. The embeddings varied in being static or contextualized, and how much affect specific information was prioritized during the pre-training and fine-tuning phase. Our analyses show that word embedding from the vanilla BERT model did not saliently encode the affect information of English words. Only when the BERT model was fine-tuned on emotion-related tasks or contained extra contextualized information from emotion-rich contexts could the corresponding embedding encode more relevant affect information.