Connecting Weighted Automata, Tensor Networks and Recurrent Neural Networks through Spectral Learning

TL;DR

This paper establishes fundamental connections between weighted finite automata, tensor networks, and recurrent neural networks, introducing a spectral learning algorithm for linear 2-RNNs that leverages low-rank Hankel structures.

Contribution

It reveals intrinsic links between WFA, tensor train decomposition, and 2-RNNs, and proposes a novel, provably consistent spectral learning algorithm for linear 2-RNNs.

Findings

Efficient spectral learning algorithm for large Hankel matrices.

Proven equivalence between WFA and 2-RNN with linear activation.

Successful validation on synthetic and real-world data.

Abstract

In this paper, we present connections between three models used in different research fields: weighted finite automata~(WFA) from formal languages and linguistics, recurrent neural networks used in machine learning, and tensor networks which encompasses a set of optimization techniques for high-order tensors used in quantum physics and numerical analysis. We first present an intrinsic relation between WFA and the tensor train decomposition, a particular form of tensor network. This relation allows us to exhibit a novel low rank structure of the Hankel matrix of a function computed by a WFA and to design an efficient spectral learning algorithm leveraging this structure to scale the algorithm up to very large Hankel matrices.We then unravel a fundamental connection between WFA and second-orderrecurrent neural networks~(2-RNN): in the case of sequences of discrete symbols, WFA and 2-RNN with linear activationfunctions are expressively equivalent. Leveraging this equivalence result combined with the classical spectral learning algorithm for weighted automata, we introduce the first provable learning algorithm for linear 2-RNN defined over sequences of continuous input vectors.This algorithm relies on estimating low rank sub-blocks of the Hankel tensor, from which the parameters of a linear 2-RNN can be provably recovered. The performances of the proposed learning algorithm are assessed in a simulation study on both synthetic and real-world data.