Combining Discrete Wavelet and Cosine Transforms for Efficient Sentence Embedding

TL;DR

This paper introduces a novel approach combining Discrete Wavelet and Cosine Transforms to create efficient, fixed-size sentence embeddings that preserve important linguistic information and improve performance on NLP tasks.

Contribution

The paper proposes a non-parameterized model using DWT and DCT for compressing sentence embeddings, demonstrating its effectiveness in NLP applications.

Findings

Wavelet transforms effectively consolidate important information in word vectors.

Combining DWT with DCT produces compact sentence embeddings.

The proposed method achieves comparable or superior results to original embeddings in downstream tasks.

Abstract

Wavelets have emerged as a cutting edge technology in a number of fields. Concrete results of their application in Image and Signal processing suggest that wavelets can be effectively applied to Natural Language Processing (NLP) tasks that capture a variety of linguistic properties. In this paper, we leverage the power of applying Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We first evaluate, intrinsically and extrinsically, how wavelets can effectively be used to consolidate important information in a word vector while reducing its dimensionality. We further combine DWT with Discrete Cosine Transform (DCT) to propose a non-parameterized model that compresses a sentence with a dense amount of information in a fixed size vector based on locally varying word features. We show the efficacy of the proposed paradigm on downstream applications models yielding comparable and even superior (in some tasks) results to original embeddings.