A Lexical Analysis Tool with Ambiguity Support

TL;DR

This paper introduces Lamb, a lexical analysis tool that generates a graph of all possible token sequences in an input, enabling context-sensitive analysis in ambiguous language scenarios.

Contribution

Lamb is the first lexical analyzer to produce a comprehensive analysis graph supporting ambiguity, enhancing parsing capabilities in complex language specifications.

Findings

Lamb effectively captures all token sequences in ambiguous inputs.

The analysis graph improves parsing accuracy in ambiguous language scenarios.

Lamb integrates seamlessly with parsers for context-sensitive analysis.

Abstract

Lexical ambiguities naturally arise in languages. We present Lamb, a lexical analyzer that produces a lexical analysis graph describing all the possible sequences of tokens that can be found within the input string. Parsers can process such lexical analysis graphs and discard any sequence of tokens that does not produce a valid syntactic sentence, therefore performing, together with Lamb, a context-sensitive lexical analysis in lexically-ambiguous language specifications.