Deconvolutional Paragraph Representation Learning

TL;DR

This paper introduces a convolutional and deconvolutional autoencoder framework for learning long text representations, outperforming RNNs in reconstructing and correcting long paragraphs, with applications in classification and summarization.

Contribution

It presents a novel purely convolutional autoencoding architecture that improves long text reconstruction and is computationally efficient, addressing RNN limitations.

Findings

Better reconstruction of long paragraphs compared to RNNs

Improved semi-supervised text classification performance

Enhanced summarization results

Abstract

Learning latent representations from long text sequences is an important first step in many natural language processing applications. Recurrent Neural Networks (RNNs) have become a cornerstone for this challenging task. However, the quality of sentences during RNN-based decoding (reconstruction) decreases with the length of the text. We propose a sequence-to-sequence, purely convolutional and deconvolutional autoencoding framework that is free of the above issue, while also being computationally efficient. The proposed method is simple, easy to implement and can be leveraged as a building block for many applications. We show empirically that compared to RNNs, our framework is better at reconstructing and correcting long paragraphs. Quantitative evaluation on semi-supervised text classification and summarization tasks demonstrate the potential for better utilization of long unlabeled text data.