Default Disambiguation for Online Parsers

TL;DR

This paper introduces an automatic language boxes algorithm that enhances online parsers by applying default disambiguation, effectively managing ambiguity in composed grammars with high accuracy in real-world language scenarios.

Contribution

It extends language boxes with default disambiguation capabilities, enabling automatic insertion, removal, or resizing of language boxes during online parsing.

Findings

Achieves 98.8% acceptability in real-world language compositions

Improves online parsing by automating ambiguity resolution

Demonstrates effectiveness over traditional delimiter-based methods

Abstract

Since composed grammars are often ambiguous, grammar composition requires a mechanism for dealing with ambiguity: either ruling it out by using delimiters (which are awkward to work with), or by using disambiguation operators to filter a parse forest down to a single parse tree (where, in general, we cannot be sure that we have covered all possible parse forests). In this paper, we show that default disambiguation, which is inappropriate for batch parsing, works well for online parsing, where it can be overridden by the user if necessary. We extend language boxes -- a delimiter-based algorithm atop incremental parsing -- in such a way that default disambiguation can automatically insert, remove, or resize, language boxes, leading to the automatic language boxes algorithm. The nature of the problem means that default disambiguation cannot always match a user's intention. However, our experimental evaluation shows that automatic language boxes behave acceptably in 98.8% of tests involving compositions of real-world programming languages.