Gaussian Mixture Latent Vector Grammars

TL;DR

This paper introduces Gaussian Mixture Latent Vector Grammars (GM-LVeGs), a flexible framework that models nonterminal subtypes with continuous vectors, enabling efficient inference and achieving competitive results in NLP tasks.

Contribution

The paper proposes GM-LVeGs, a novel extension of latent vector grammars using Gaussian mixtures, allowing efficient inference and broadening the modeling capabilities of grammar-based NLP methods.

Findings

Achieves competitive accuracy in part-of-speech tagging.

Enables efficient inference through an extended inside-outside algorithm.

Unifies previous latent variable models as special cases.

Abstract

We introduce Latent Vector Grammars (LVeGs), a new framework that extends latent variable grammars such that each nonterminal symbol is associated with a continuous vector space representing the set of (infinitely many) subtypes of the nonterminal. We show that previous models such as latent variable grammars and compositional vector grammars can be interpreted as special cases of LVeGs. We then present Gaussian Mixture LVeGs (GM-LVeGs), a new special case of LVeGs that uses Gaussian mixtures to formulate the weights of production rules over subtypes of nonterminals. A major advantage of using Gaussian mixtures is that the partition function and the expectations of subtype rules can be computed using an extension of the inside-outside algorithm, which enables efficient inference and learning. We apply GM-LVeGs to part-of-speech tagging and constituency parsing and show that GM-LVeGs can achieve competitive accuracies. Our code is available at https://github.com/zhaoyanpeng/lveg.