Adaptive Recurrent Neural Network Based on Mixture Layer

TL;DR

This paper introduces an adaptive RNN with a mixture layer that clusters sequence patterns into prototypes, enhancing its ability to model sequences with multiple patterns, and demonstrates its effectiveness through experiments.

Contribution

The paper proposes a novel mixture layer for RNNs that adaptively updates states based on pattern prototypes, improving multi-pattern sequence modeling.

Findings

Effective in modeling sequences with multiple patterns

Improves RNN performance on synthetic and real datasets

Can incorporate prior knowledge for further enhancement

Abstract

Although Recurrent Neural Network (RNN) has been a powerful tool for modeling sequential data, its performance is inadequate when processing sequences with multiple patterns. In this paper, we address this challenge by introducing a novel mixture layer and constructing an adaptive RNN. The mixture layer augmented RNN (termed as M-RNN) partitions patterns in training sequences into several clusters and stores the principle patterns as prototype vectors of components in a mixture model. By leveraging the mixture layer, the proposed method can adaptively update states according to the similarities between encoded inputs and prototype vectors, leading to a stronger capacity in assimilating sequences with multiple patterns. Moreover, our approach can be further extended by taking advantage of prior knowledge about data. Experiments on both synthetic and real datasets demonstrate the effectiveness of the proposed method.