PCFGs Can Do Better: Inducing Probabilistic Context-Free Grammars with Many Symbols

TL;DR

This paper introduces a tensor decomposition-based parameterization for probabilistic context-free grammars that reduces computational complexity, enabling the use of more symbols and improving unsupervised parsing across multiple languages.

Contribution

It proposes a novel tensor decomposition approach for PCFGs that scales better and enhances unsupervised grammar induction with neural parameterization.

Findings

More symbols lead to better parsing performance

Tensor decomposition reduces computational complexity

Model effective across ten languages

Abstract

Probabilistic context-free grammars (PCFGs) with neural parameterization have been shown to be effective in unsupervised phrase-structure grammar induction. However, due to the cubic computational complexity of PCFG representation and parsing, previous approaches cannot scale up to a relatively large number of (nonterminal and preterminal) symbols. In this work, we present a new parameterization form of PCFGs based on tensor decomposition, which has at most quadratic computational complexity in the symbol number and therefore allows us to use a much larger number of symbols. We further use neural parameterization for the new form to improve unsupervised parsing performance. We evaluate our model across ten languages and empirically demonstrate the effectiveness of using more symbols. Our code: https://github.com/sustcsonglin/TN-PCFG