Pearl: A Probabilistic Chart Parser

TL;DR

Pearl is a probabilistic chart parser that uses context-rich conditional probabilities to improve natural language parsing accuracy, integrating various linguistic models into a unified framework.

Contribution

It introduces Pearl, a novel probabilistic chart parsing algorithm that combines bottom-up and top-down predictions with context-based scoring, unlike previous stochastic parsers.

Findings

Successfully resolves part-of-speech ambiguity

Determines categories for unknown words

Selects correct parses with a loose grammar

Abstract

This paper describes a natural language parsing algorithm for unrestricted text which uses a probability-based scoring function to select the "best" parse of a sentence. The parser, Pearl, is a time-asynchronous bottom-up chart parser with Earley-type top-down prediction which pursues the highest-scoring theory in the chart, where the score of a theory represents the extent to which the context of the sentence predicts that interpretation. This parser differs from previous attempts at stochastic parsers in that it uses a richer form of conditional probabilities based on context to predict likelihood. Pearl also provides a framework for incorporating the results of previous work in part-of-speech assignment, unknown word models, and other probabilistic models of linguistic features into one parsing tool, interleaving these techniques instead of using the traditional pipeline architecture. In preliminary tests, Pearl has been successful at resolving part-of-speech and word (in speech processing) ambiguity, determining categories for unknown words, and selecting correct parses first using a very loosely fitting covering grammar.