Transformation of Dense and Sparse Text Representations

TL;DR

This paper introduces a Semantic Transformation method inspired by Fourier Transform to convert dense text representations into sparse forms, enabling new NLP research avenues and improving downstream task performance.

Contribution

It proposes a novel transformation technique that bridges dense and sparse representations, allowing flexible use of both in NLP tasks.

Findings

Effective in transforming dense to sparse representations

Improves performance on classification and inference tasks

Enables leveraging sparsity for explainability

Abstract

Sparsity is regarded as a desirable property of representations, especially in terms of explanation. However, its usage has been limited due to the gap with dense representations. Most NLP research progresses in recent years are based on dense representations. Thus the desirable property of sparsity cannot be leveraged. Inspired by Fourier Transformation, in this paper, we propose a novel Semantic Transformation method to bridge the dense and sparse spaces, which can facilitate the NLP research to shift from dense space to sparse space or to jointly use both spaces. The key idea of the proposed approach is to use a Forward Transformation to transform dense representations to sparse representations. Then some useful operations in the sparse space can be performed over the sparse representations, and the sparse representations can be used directly to perform downstream tasks such as text classification and natural language inference. Then, a Backward Transformation can also be carried out to transform those processed sparse representations to dense representations. Experiments using classification tasks and natural language inference task show that the proposed Semantic Transformation is effective.