Discrete Cosine Transform as Universal Sentence Encoder

TL;DR

This paper demonstrates that Discrete Cosine Transform (DCT) can serve as an effective universal sentence encoder across multiple languages, outperforming traditional averaging methods on various datasets.

Contribution

The study extends DCT-based sentence encoding to multiple languages, showing its effectiveness beyond English and establishing it as a versatile universal encoder.

Findings

DCT encoder outperforms averaging in multilingual settings

Consistent performance improvements across datasets

Effective for languages like German, French, Spanish, and Russian

Abstract

Modern sentence encoders are used to generate dense vector representations that capture the underlying linguistic characteristics for a sequence of words, including phrases, sentences, or paragraphs. These kinds of representations are ideal for training a classifier for an end task such as sentiment analysis, question answering and text classification. Different models have been proposed to efficiently generate general purpose sentence representations to be used in pretraining protocols. While averaging is the most commonly used efficient sentence encoder, Discrete Cosine Transform (DCT) was recently proposed as an alternative that captures the underlying syntactic characteristics of a given text without compromising practical efficiency compared to averaging. However, as with most other sentence encoders, the DCT sentence encoder was only evaluated in English. To this end, we utilize DCT encoder to generate universal sentence representation for different languages such as German, French, Spanish and Russian. The experimental results clearly show the superior effectiveness of DCT encoding in which consistent performance improvements are achieved over strong baselines on multiple standardized datasets.