Whitening Not Recommended for Classification Tasks in LLMs

TL;DR

This paper investigates the effects of whitening operations on sentence embeddings from large language models, revealing that whitening can harm classification performance and is not universally beneficial.

Contribution

The study provides a comprehensive analysis showing whitening's negative impact on classification tasks in LLM embeddings and introduces SentEval+ for embedding evaluation.

Findings

Whitening can degenerate embeddings for classification tasks.

Effectiveness of whitening is model- and task-dependent.

Introduces SentEval+ platform for embedding evaluation.

Abstract

Sentence embedding is a cornerstone in NLP. Whitening has been claimed to be an effective operation to improve embedding quality obtained from Large Language Models (LLMs). However, we find that the efficacy of whitening is model-dependent and task-dependent. In particular, whitening degenerates embeddings for classification tasks. The conclusion is supported by extensive experiments. We also explored a variety of whitening operations, including PCA, ZCA, PCA-Cor, ZCA-Cor and Cholesky whitenings. A by-product of our research is embedding evaluation platform for LLMs called SentEval+.