Static Word Embeddings for Sentence Semantic Representation

TL;DR

This paper introduces optimized static word embeddings derived from pre-trained models and enhanced with sentence-level analysis, achieving superior performance in sentence semantic tasks with low computational cost.

Contribution

It presents a novel method combining sentence-level PCA and knowledge distillation or contrastive learning to improve static word embeddings for sentence semantics.

Findings

Outperforms existing static models on semantic tasks

Surpasses basic Sentence Transformer (SimCSE) on embedding benchmarks

Effectively removes irrelevant embedding components and adjusts vector norms

Abstract

We propose new static word embeddings optimised for sentence semantic representation. We first extract word embeddings from a pre-trained Sentence Transformer, and improve them with sentence-level principal component analysis, followed by either knowledge distillation or contrastive learning. During inference, we represent sentences by simply averaging word embeddings, which requires little computational cost. We evaluate models on both monolingual and cross-lingual tasks and show that our model substantially outperforms existing static models on sentence semantic tasks, and even surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. Lastly, we perform a variety of analyses and show that our method successfully removes word embedding components that are not highly relevant to sentence semantics, and adjusts the vector norms based on the influence of words on sentence semantics.