Semisupervised Autoencoder for Sentiment Analysis

TL;DR

This paper introduces a semisupervised autoencoder that incorporates supervision into the loss function to improve sentiment analysis, especially with high-dimensional textual data and limited labeled samples.

Contribution

The paper proposes a novel autoencoder training method using classifier weights and Bayesian marginalization, enhancing scalability and discriminative feature learning for sentiment analysis.

Findings

Outperforms existing methods on six sentiment datasets

Effectively leverages unlabeled data for improved accuracy

Learns highly discriminative feature representations

Abstract

In this paper, we investigate the usage of autoencoders in modeling textual data. Traditional autoencoders suffer from at least two aspects: scalability with the high dimensionality of vocabulary size and dealing with task-irrelevant words. We address this problem by introducing supervision via the loss function of autoencoders. In particular, we first train a linear classifier on the labeled data, then define a loss for the autoencoder with the weights learned from the linear classifier. To reduce the bias brought by one single classifier, we define a posterior probability distribution on the weights of the classifier, and derive the marginalized loss of the autoencoder with Laplace approximation. We show that our choice of loss function can be rationalized from the perspective of Bregman Divergence, which justifies the soundness of our model. We evaluate the effectiveness of our model on six sentiment analysis datasets, and show that our model significantly outperforms all the competing methods with respect to classification accuracy. We also show that our model is able to take advantage of unlabeled dataset and get improved performance. We further show that our model successfully learns highly discriminative feature maps, which explains its superior performance.