TextCNN with Attention for Text Classification

TL;DR

This paper enhances TextCNN for text classification by integrating an attention mechanism and introducing WordRank for vocabulary selection, resulting in improved accuracy and reduced model size.

Contribution

It proposes an attention-augmented TextCNN and a novel WordRank method for vocabulary selection, reducing parameters with minimal accuracy loss.

Findings

TextCNN accuracy on 20News increased from 94.79% to 96.88%.

Vocabulary size reduced by over 5x with only 1.2% accuracy decrease.

Model training is faster due to fewer parameters.

Abstract

The vast majority of textual content is unstructured, making automated classification an important task for many applications. The goal of text classification is to automatically classify text documents into one or more predefined categories. Recently proposed simple architectures for text classification such as Convolutional Neural Networks for Sentence Classification by Kim, Yoon showed promising results. In this paper, we propose incorporating an attention mechanism into the network to boost its performance, we also propose WordRank for vocabulary selection to reduce the network embedding parameters and speed up training with minimum accuracy loss. By adopting the proposed ideas TextCNN accuracy on 20News increased from 94.79 to 96.88, moreover, the number of parameters for the embedding layer can be reduced substantially with little accuracy loss by using WordRank. By using WordRank for vocabulary selection we can reduce the number of parameters by more than 5x from 7.9M to 1.5M, and the accuracy will only decrease by 1.2%.