Low-Rank Approximation of Weighted Tree Automata

TL;DR

This paper introduces an efficient method for minimizing weighted tree automata using singular value decomposition of their Hankel matrices, leading to improved model stability and lower perplexity in language modeling tasks.

Contribution

The paper presents a novel algorithm for SVD of infinite Hankel matrices represented by WTA, enabling effective minimization and approximation of weighted tree automata.

Findings

Achieves lower perplexity than previous PCFG minimization methods.

Demonstrates increased stability due to absence of local optima.

Validates effectiveness on real-world newswire treebank data.

Abstract

We describe a technique to minimize weighted tree automata (WTA), a powerful formalisms that subsumes probabilistic context-free grammars (PCFGs) and latent-variable PCFGs. Our method relies on a singular value decomposition of the underlying Hankel matrix defined by the WTA. Our main theoretical result is an efficient algorithm for computing the SVD of an infinite Hankel matrix implicitly represented as a WTA. We provide an analysis of the approximation error induced by the minimization, and we evaluate our method on real-world data originating in newswire treebank. We show that the model achieves lower perplexity than previous methods for PCFG minimization, and also is much more stable due to the absence of local optima.