Weighted Parsing for Grammar-Based Language Models over Multioperator Monoids

TL;DR

This paper introduces a comprehensive framework for weighted parsing using multioperator monoids, extending existing models and applicable to diverse areas like algebraic dynamic programming, with proven algorithm correctness and termination.

Contribution

It presents a novel, generalized approach to weighted parsing based on multioperator monoids, broadening the scope beyond traditional semiring-based models.

Findings

Algorithm for weighted parsing developed and proven correct.

Framework generalizes previous models including semiring parsing.

Ensures termination for a large class of grammar-based models.

Abstract

We develop a general framework for weighted parsing which is built on top of grammar-based language models and employs multioperator monoids as weight algebras. It generalizes previous work in that area (semiring parsing, weighted deductive parsing) and also covers applications outside the classical scope of parsing, e.g., algebraic dynamic programming. We show an algorithm for weighted parsing and, for a large class of weighted grammar-based language models, we prove formally that it terminates and is correct.