Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling

TL;DR

The paper introduces Lattice Recurrent Units (LRU), a novel recurrent model architecture that improves convergence and efficiency in sequence modeling, especially under resource constraints, by decoupling information flow along time and depth dimensions.

Contribution

It proposes a new LRU architecture with coupled flows along time and depth, and analyzes the effects of decoupling key components, demonstrating improved convergence and statistical efficiency.

Findings

LRU models outperform Grid-LSTM and Recurrent Highway networks in convergence rates.

LRU achieves higher statistical efficiency in sequence learning tasks.

LRU produces more accurate language models on benchmark datasets.

Abstract

Recurrent neural networks have shown remarkable success in modeling sequences. However low resource situations still adversely affect the generalizability of these models. We introduce a new family of models, called Lattice Recurrent Units (LRU), to address the challenge of learning deep multi-layer recurrent models with limited resources. LRU models achieve this goal by creating distinct (but coupled) flow of information inside the units: a first flow along time dimension and a second flow along depth dimension. It also offers a symmetry in how information can flow horizontally and vertically. We analyze the effects of decoupling three different components of our LRU model: Reset Gate, Update Gate and Projected State. We evaluate this family on new LRU models on computational convergence rates and statistical efficiency. Our experiments are performed on four publicly-available datasets, comparing with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has better empirical computational convergence rates and statistical efficiency values, along with learning more accurate language models.