A Lexical Analysis Tool with Ambiguity Support
Luis Quesada, Fernando Berzal, Francisco J. Cortijo

TL;DR
This paper introduces Lamb, a lexical analysis tool that generates a graph of all possible token sequences in an input, enabling context-sensitive analysis in ambiguous language scenarios.
Contribution
Lamb is the first lexical analyzer to produce a comprehensive analysis graph supporting ambiguity, enhancing parsing capabilities in complex language specifications.
Findings
Lamb effectively captures all token sequences in ambiguous inputs.
The analysis graph improves parsing accuracy in ambiguous language scenarios.
Lamb integrates seamlessly with parsers for context-sensitive analysis.
Abstract
Lexical ambiguities naturally arise in languages. We present Lamb, a lexical analyzer that produces a lexical analysis graph describing all the possible sequences of tokens that can be found within the input string. Parsers can process such lexical analysis graphs and discard any sequence of tokens that does not produce a valid syntactic sentence, therefore performing, together with Lamb, a context-sensitive lexical analysis in lexically-ambiguous language specifications.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Algorithms and Data Compression
