Neural Network Based Nonlinear Weighted Finite Automata

TL;DR

This paper introduces a neural network-based nonlinear extension of weighted finite automata (WFA), demonstrating improved modeling capacity and efficiency in learning complex string functions from data.

Contribution

It proposes a novel NL-WFA model with a spectral-inspired learning algorithm using auto-encoders, advancing the capabilities of finite automata in machine learning.

Findings

NL-WFA can model complex grammatical structures.

The model achieves smaller sizes compared to linear WFA.

Effective on both synthetic and real-world data.

Abstract

Weighted finite automata (WFA) can expressively model functions defined over strings but are inherently linear models. Given the recent successes of nonlinear models in machine learning, it is natural to wonder whether ex-tending WFA to the nonlinear setting would be beneficial. In this paper, we propose a novel model of neural network based nonlinearWFA model (NL-WFA) along with a learning algorithm. Our learning algorithm is inspired by the spectral learning algorithm for WFAand relies on a nonlinear decomposition of the so-called Hankel matrix, by means of an auto-encoder network. The expressive power of NL-WFA and the proposed learning algorithm are assessed on both synthetic and real-world data, showing that NL-WFA can lead to smaller model sizes and infer complex grammatical structures from data.