Rational Recurrences
Hao Peng, Roy Schwartz, Sam Thomson, and Noah A. Smith

TL;DR
This paper explores the connection between recurrent neural networks and weighted finite state automata (WFSAs), introducing rational recurrences that leverage this link to improve language modeling and text classification.
Contribution
It formally characterizes rational recurrences as WFSA-based hidden state updates and demonstrates their application in designing improved neural architectures.
Findings
Several neural models use rational recurrences.
The new model outperforms recent baselines on language tasks.
Transferring classical automata insights enhances neural model design.
Abstract
Despite the tremendous empirical success of neural models in natural language processing, many of them lack the strong intuitions that accompany classical machine learning approaches. Recently, connections have been shown between convolutional neural networks (CNNs) and weighted finite state automata (WFSAs), leading to new interpretations and insights. In this work, we show that some recurrent neural networks also share this connection to WFSAs. We characterize this connection formally, defining rational recurrences to be recurrent hidden state update functions that can be written as the Forward calculation of a finite set of WFSAs. We show that several recent neural models use rational recurrences. Our analysis provides a fresh view of these models and facilitates devising new neural architectures that draw inspiration from WFSAs. We present one such model, which performs better than…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Machine Learning and Algorithms · Natural Language Processing Techniques
