Hierarchical Multiscale Recurrent Neural Networks

TL;DR

This paper introduces a hierarchical multiscale recurrent neural network that captures latent hierarchical structures in sequences by encoding dependencies at multiple timescales, demonstrated on language and handwriting tasks.

Contribution

It proposes a novel multiscale RNN architecture with a new update mechanism to discover hierarchical sequence structures without explicit boundary cues.

Findings

Effectively captures hierarchical sequence structures

Outperforms baseline models in language modeling

Successfully models handwriting sequences

Abstract

Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural networks, which can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that our proposed multiscale architecture can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence modelling.