Embedding Compression for Text Classification Using Dictionary Screening

TL;DR

This paper introduces a dictionary screening technique for embedding compression in text classification, significantly reducing model size while maintaining high prediction accuracy.

Contribution

It presents a novel importance assessment method for keywords, enabling effective dictionary screening and embedding compression in text classification models.

Findings

Significant reduction in dictionary size and text sequence length.

Maintains competitive prediction accuracy after compression.

Demonstrated effectiveness through extensive numerical studies.

Abstract

In this paper, we propose a dictionary screening method for embedding compression in text classification tasks. The key purpose of this method is to evaluate the importance of each keyword in the dictionary. To this end, we first train a pre-specified recurrent neural network-based model using a full dictionary. This leads to a benchmark model, which we then use to obtain the predicted class probabilities for each sample in a dataset. Next, to evaluate the impact of each keyword in affecting the predicted class probabilities, we develop a novel method for assessing the importance of each keyword in a dictionary. Consequently, each keyword can be screened, and only the most important keywords are reserved. With these screened keywords, a new dictionary with a considerably reduced size can be constructed. Accordingly, the original text sequence can be substantially compressed. The proposed method leads to significant reductions in terms of parameters, average text sequence, and dictionary size. Meanwhile, the prediction power remains very competitive compared to the benchmark model. Extensive numerical studies are presented to demonstrate the empirical performance of the proposed method.