On Multiplicative Integration with Recurrent Neural Networks

TL;DR

This paper proposes Multiplicative Integration (MI), a simple structural modification for RNNs that enhances information flow and improves performance across various models and tasks with minimal additional parameters.

Contribution

It introduces MI as a versatile, easy-to-implement structural change that can be integrated into existing RNN architectures like LSTMs and GRUs.

Findings

MI significantly improves RNN performance on multiple tasks.

MI can be embedded into various RNN models with minimal parameter increase.

Empirical analysis shows better learning behavior with MI.

Abstract

We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.