Discrete Autoencoders for Sequence Models

TL;DR

This paper introduces a novel sequence modeling approach that incorporates a discrete autoencoder with semantic hashing to improve representation quality, interpretability, and diversity in tasks like language modeling and translation.

Contribution

It proposes a new discrete autoencoder with semantic hashing for sequence models, enhancing representation learning and enabling diverse translation generation.

Findings

Semantic hashing improves gradient propagation in discrete latent spaces.

Latent codes correspond to meaningful words and phrases.

Autoencoder-augmented models generate more diverse translations.

Abstract

Recurrent models for sequences have been recently successful at many tasks, especially for language modeling and machine translation. Nevertheless, it remains challenging to extract good representations from these models. For instance, even though language has a clear hierarchical structure going from characters through words to sentences, it is not apparent in current language models. We propose to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space. In order to propagate gradients though this discrete representation we introduce an improved semantic hashing technique. We show that this technique performs well on a newly proposed quantitative efficiency measure. We also analyze latent codes produced by the model showing how they correspond to words and phrases. Finally, we present an application of the autoencoder-augmented model to generating diverse translations.