Nez: practical open grammar language

TL;DR

Nez is a practical, open, PEG-based grammar language that enables reuse across programming languages, supporting complex syntax constructs and achieving competitive parser performance through innovative extensions and virtual machine implementation.

Contribution

Nez introduces extensions for context-sensitive syntax recognition and flexible AST construction, enabling many grammars to be parsed without action code.

Findings

Nez can parse many complex language grammars.

Nez parsers achieve performance comparable to existing parsers.

Nez's virtual machine approach enhances parser efficiency.

Abstract

Nez is a PEG(Parsing Expressing Grammar)-based open grammar language that allows us to describe complex syntax constructs without action code. Since open grammars are declarative and free from a host programming language of parsers, software engineering tools and other parser applications can reuse once-defined grammars across programming languages. A key challenge to achieve practical open grammars is the expressiveness of syntax constructs and the resulting parser performance, as the traditional action code approach has provided very pragmatic solutions to these two issues. In Nez, we extend the symbol-based state management to recognize context-sensitive language syntax, which often appears in major programming languages. In addition, the Abstract Syntax Tree constructor allows us to make flexible tree structures, including the left-associative pair of trees. Due to these extensions, we have demonstrated that Nez can parse not all but many grammars. Nez can generate various types of parsers since all Nez operations are independent of a specific parser language. To highlight this feature, we have implemented Nez with dynamic parsing, which allows users to integrate a Nez parser as a parser library that loads a grammar at runtime. To achieve its practical performance, Nez operators are assembled into low-level virtual machine instructions, including automated state modifications when backtracking, transactional controls of AST construction, and efficient memoization in packrat parsing. We demonstrate that Nez dynamic parsers achieve very competitive performance compared to existing efficient parser generators.