Rational Recurrences

TL;DR

This paper explores the connection between recurrent neural networks and weighted finite state automata (WFSAs), introducing rational recurrences that leverage this link to improve language modeling and text classification.

Contribution

It formally characterizes rational recurrences as WFSA-based hidden state updates and demonstrates their application in designing improved neural architectures.

Findings

Several neural models use rational recurrences.

The new model outperforms recent baselines on language tasks.

Transferring classical automata insights enhances neural model design.

Abstract

Despite the tremendous empirical success of neural models in natural language processing, many of them lack the strong intuitions that accompany classical machine learning approaches. Recently, connections have been shown between convolutional neural networks (CNNs) and weighted finite state automata (WFSAs), leading to new interpretations and insights. In this work, we show that some recurrent neural networks also share this connection to WFSAs. We characterize this connection formally, defining rational recurrences to be recurrent hidden state update functions that can be written as the Forward calculation of a finite set of WFSAs. We show that several recent neural models use rational recurrences. Our analysis provides a fresh view of these models and facilitates devising new neural architectures that draw inspiration from WFSAs. We present one such model, which performs better than two recent baselines on language modeling and text classification. Our results demonstrate that transferring intuitions from classical models like WFSAs can be an effective approach to designing and understanding neural models.