An Attention-Gated Convolutional Neural Network for Sentence Classification

TL;DR

This paper introduces AGCNN, an attention-gated CNN model that effectively utilizes limited contextual information for sentence classification, achieving higher accuracy and demonstrating the benefits of a novel activation function.

Contribution

The paper presents a novel attention-gated CNN architecture with a new activation function, improving sentence classification accuracy over standard models.

Findings

AGCNN achieves up to 3.1% higher accuracy than standard CNNs.

NLReLU activation outperforms ReLU and is competitive with other functions.

Model shows strong performance on multiple sentence classification tasks.

Abstract

The classification of sentences is very challenging, since sentences contain the limited contextual information. In this paper, we proposed an Attention-Gated Convolutional Neural Network (AGCNN) for sentence classification, which generates attention weights from the feature's context windows of different sizes by using specialized convolution encoders. It makes full use of limited contextual information to extract and enhance the influence of important features in predicting the sentence's category. Experimental results demonstrated that our model can achieve up to 3.1% higher accuracy than standard CNN models, and gain competitive results over the baselines on four out of the six tasks. Besides, we designed an activation function, namely, Natural Logarithm rescaled Rectified Linear Unit (NLReLU). Experiments showed that NLReLU can outperform ReLU and is comparable to other well-known activation functions on AGCNN.