CPEG: A Typed Tree Construction from Parsing Expression Grammars with Regex-Like Captures

TL;DR

This paper introduces CPEG, an extended parsing expression grammar with regex-like captures that constructs syntax trees with guaranteed structural constraints, reducing user effort and enabling formal type inference.

Contribution

It presents a novel CPEG formalism with capture annotations and a formal type inference system ensuring structural constraints and soundness.

Findings

CPEG guarantees structural constraints of syntax trees for any input.

Type inference for CPEG is sound and unique.

CPEG reduces user code for syntax validation.

Abstract

CPEG is an extended parsing expression grammar with regex-like capture annotation. Two annotations (capture and left-folding) allow a flexible construction of syntax trees from arbitrary parsing patterns. More importantly, CPEG is designed to guarantee structural constraints of syntax trees for any input strings. This reduces the amount of user code needed to check whether the intended elements exist. To represent the structural constraints, we focus on regular expression types, a variant formalism of tree automata, which have been intensively studied in the context of XML schemas. Regular expression type is inferred from a given CPEG by the type inference that is formally developed in this paper. We prove the soundness and the uniqueness of the type inference. The type inference enables a CPEG to serve both as a syntactic specification of the input and a schematic specification of the output.

Tables

Table 1. Table 1. PEG operators

PEG	Type	Description
’ ’	Primary	Matches text
$\left[\right]$	Primary	Matches character class
.	Primary	Any character
$A$	Primary	Nonterminal application
$\left(e\right)$	Primary	Grouping
$e\text{?}$	Unary suffix	Option
$e\ast$	Unary suffix	Zero-or-more repetitions
$e\text{+}$	Unary suffix	One-or-more repetitions
	Unary prefix	And-predicate
$\text{!}e$	Unary prefix	Negation
$e_1{e}_2$	Binary	Sequencing
$e_1\text{/}{e}_2$	Binary	Prioritized Choice

Equations

$$A={\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}$$

$$\mathtt{type}\mathtt{ProdM}$$

$$\mathtt{type}\mathtt{Val}$$

$$\mathtt{type}\mathtt{ProdL}$$

$$\mathtt{type}\mathtt{Val}$$

$$\begin{array}[]{lllll}e&::=&\varepsilon&&\mbox{empty}\\ &&a&(a\in\Sigma)&\mbox{terminal}\\ &&A&(A\in N_{G})&\mbox{nonterminal}\\ &&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}{\;}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}&&\mbox{sequence}\\ &&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\;\texttt{/}\;}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}&&\mbox{ordered choice}\\ &&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\ast}&&\mbox{repetition}\\ &&{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{!}}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}&&\mbox{not-predicate}\\ &&\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}&({\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\in\mathcal{S})&\mbox{capture}\\ &&{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge\kern-3.00003pt{{}^{\ast}}}\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}}&({\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\in\mathcal{S})&\mbox{fold-capture}\\ \end{array}$$

$${}^{\prime }ab{c}^{\prime }$$

$$\left[abc\right]$$

$$\displaystyle{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{+}}$$

$$\displaystyle{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{?}}$$

$$\displaystyle{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{\&}}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}$$

$$\displaystyle{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}\wedge\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}$$

$$\displaystyle{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}(\wedge\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}){\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\;\texttt{/}\;}(\wedge\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{3}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L^{\prime}}}\})$$

$$\displaystyle{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}(\wedge\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}){\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{?}}$$

$$A\leftarrow\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L_{1}}}\}A\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L_{2}}}\}~{}/~{}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}$$

$$A\leftarrow\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L_{1}}}\}A~{}/~{}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}$$

$$A\leftarrow\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L_{1}}}\}Aa~{}/~{}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}$$

$$A$$

$$B$$

$$\begin{array}[]{lllll}v&::=&{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v]&({\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\in\mathcal{S}){}&\mbox{node}\\ &&v_{1},v_{2}&&\mbox{concatenation}\\ &&x&(x\in\Sigma^{\ast})&\mbox{string}\\ \end{array}$$

$${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}[{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{123}],{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{45}],{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{6}]]$$

$${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}[{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}[{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{123}],{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{45}]],{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{6}]]$$

$${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[\underbrace{{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[\dots\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}[{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v_{1}}_{n-1},v_{2}],v_{3}],\dots,v_{n-1}],v_{n}].$$

$$\begin{array}[]{llll}{\mathtt{T}}&::=&\mathtt{Empty}&\mbox{empty sequence}\\ &&\mathtt{T_{1},T_{2}}&\mbox{concatenation}\\ &&\mathtt{T_{1}}~{}|~{}\mathtt{T_{2}}&\mbox{union}\\ &&\mathtt{T^{*}}&\mbox{repetition}\\ &&\mathtt{{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[T_{1},\dots,T_{n}]}&\mbox{label}\\ &&\mathtt{X}&\mbox{type variable}\\ \end{array}$$

$$\mathtt{type}\mathtt{X}=\mathtt{T}$$

$$\mathtt{S}<:\mathtt{T}⟺\forall v\in {\mathcal{T}}_{\mathcal{S}\times {\Sigma }^{\ast }}.v:\mathtt{S}\Rightarrow v:\mathtt{T}$$

$$G=$$

$$P_G=$$

$$\displaystyle Val\leftarrow\{{[0-9]}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}\}\}$$

$$E$$

$$E_1$$

$$E_2$$

$$E_3$$

$$Prod:{\mathtt{X}}_1⊢\text{APLstar}Val:\mathtt{Empty},{\mathtt{X}}_4{|}_{\{{\mathtt{X}}_4\}}{E}_3$$

$$Prod:\mathtt{X_{1}}\vdash{{Val}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge\kern-3.00003pt{{}^{\ast}}}\{{~{}\APLstar~{}Val}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Prod}}\}}:\mathtt{X_{2}}~{}|_{\{\mathtt{X_{2},X_{3},X_{4}}\}}~{}E_{1}$$

$$\left\{\begin{array}[]{llll}\mathtt{type~{}X_{1}}=\mathtt{X_{2}},&\mathtt{type~{}X_{2}}={\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Prod}}[\mathtt{X_{2}},~{}\mathtt{Empty},\mathtt{X_{4}}]~{}|~{}\mathtt{X_{3}},\\ \mathtt{type~{}X_{3}}={\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{Empty}],&\mathtt{type~{}X_{4}}={\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{Empty}]\end{array}\right\}$$

$$\forall v\in \{v|\exists x,y.e_s{\Downarrow }_y^{x}v\}.\Gamma ⊢e_s:\mathtt{T}{|}_{\chi }E\Rightarrow v:\mathtt{T}$$

Full text

To represent the structural constraints, we focus on regular expression types, a variant formalism of tree automata, which have been intensively studied in the context of XML schemas. Regular expression type is inferred from a given CPEG by the type inference that is formally developed in this paper. We prove the soundness and the uniqueness of the type inference. The type inference enables a CPEG to serve both as a syntactic specification of the input and a schematic specification of the output.

Parsing, parsing expression grammars, type inference, regular expression types

††journalyear: 2019††copyright: acmlicensed††conference: The 34th ACM/SIGAPP Symposium on Applied Computing; April 8–12, 2019; Limassol, Cyprus††booktitle: The 34th ACM/SIGAPP Symposium on Applied Computing (SAC ’19), April 8–12, 2019, Limassol, Cyprus††price: 15.00††doi: 10.1145/3297280.3297433††isbn: 978-1-4503-5933-7/19/04††ccs: Software and its engineering Syntax††ccs: Software and its engineering Parsers

1. Introduction

Regular expressions, or regexes, are a popular language tool that describe a complex text pattern and can match some set of strings. Its popularity, however, results not only from excellent pattern matching but also declarative data extraction, in which any sub-pattern within parentheses will be captured as numbered groups. Since the capture capability provides a straightforward means for extracting parsed data, many small parsers have been implemented simply with the help of regexes.

The aim of our study is to bring the regex-like capture capability into parsing expression grammars, or PEGs (Ford, 2004). The reason for this aim is that PEGs are more expressive than regular expressions, which would enable us to extract some nested data (such as XML and JSON) that cannot be recognized by regexes. On the other hand, the capture capability would provide PEGs with a more straightforward means for implementing a parser than existing parser generators with action code.

Nez parser (Kuramitsu, 2016) is based on a declaratively extended parsing expression grammar with regex-like capture annotation. Two extended annotations (capture and left-folding) allow a flexible construction of syntax trees from arbitrary parsing patterns. We have demonstrated that Nez can parse many programming languages including Java, JavaScript, and Python.

While the declarative tree construction in Nez is convenient, it is still weak since the constructed trees are untyped. This means that they are treated as a common structure of tree data. To traverse their contents safely, the users need to check whether the intended elements exist. Parser generators such as Yacc (Johnson, 1975) and ANTLR (Parr and Fisher, 2011) can produce typed trees throughout their programmed action code, which seems more suitable for handling complex syntax trees. Similarly, typed trees, or trees whose structures are well guaranteed, are desirable in the declarative tree construction.

The main challenge of this paper is to provide typing rules for syntax trees that will be captured by a PEG. A critical issue is to infer a type of tree from a grammar, before constructing concrete trees. This indicates that a declarative grammar guarantees some structural constraints of parse trees of any input strings. This property could be a good foundation for further static binding with a programming language.

As the first attempt to infer a type of tree, we carefully designed CPEG to produce labeled unranked trees, which are equal to XML documents (Hosoya, 2010) (Note that CPEGs are a substantial subset of the Nez grammars). Here, we focus on regular expression types, or RETs, which have been studied in a foundation of XML schemas and a type system of tree automata (Hosoya and Pierce, 2003). The type inference for CPEGs developed in this paper shows that CPEGs become a schematic specification of the output and not only a syntactic specification of the input.

The remainder of this paper proceeds as follows. In Section 2, we describe our motivation using Nez grammar and RETs. In Section 3, we formally develop a CPEG as a string-to-tree transducer. In Section 4, we introduce regular expression types and define type inference rules for CPEG. In Section 5, we prove the soundness and the uniqueness of our type inference. In Section 6, we review related work, and Section 7 concludes this paper.

2. Motivating Examples

We describe a motivation of tree construction and its typing in the context of PEGs. Here we use Nez grammar (Kuramitsu, 2015), which is an open source implementation of CPEG.

2.1. Parsing Expression Grammars

Nez grammar is a PEG-based grammar specification language, whose constructs come from those of PEGs. Nez grammar is a set of syntax rules that are defined by a mapping from a non-terminal $A$ to a parsing expression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}:

[TABLE]

Table 1 shows a list of PEG operators used in Nez which inherits the formal interpretation of PEGs (Ford, 2004). This indicates that the string ’abc’ exactly matches the same input, while [abc] matches one of these characters. The . operator matches any single character. The {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{?}}, {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\ast}, and {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{+}} expressions behave as in common regular expressions, except that they are greedy and match until the longest position. The {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}{\;}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2} attempts two expressions {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1} and {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2} sequentially, backtracking to the starting position if either expression fails. The choice {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\;\texttt{/}\;}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2} first attempt {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1} and then attempt {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2} if {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1} fails. The expression {\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{\&}}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} attempts {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} without consuming any character. The expression {\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\texttt{!}}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} succeeds if {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} fails but fails if {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} succeeds.

In general, PEGs can express all languages that can be expressed by deterministic context-free grammars (such as LALR and LL($k$) grammars).

PEGs, on the other hand, provide no specification for the output of a parser, while the parser users require certain forms of syntax trees that contain all necessary information for further processing. Note that we can regard non-terminals as a tree constructor in a way that a labeled tree node $A[...]$ is constructed from A={\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}. This approach is similar to that of derivation trees, resulting in redundantly nested trees. Besides, some forms of trees are not well constructed, as described in Section 2.2.2.

2.2. Tree Annotation

PEG only provides the syntactic matching capability while Nez provides two additional annotations (called capture and fold-capture) to construct complex syntax trees in a parser context.

2.2.1. Capture

The capture annotation \{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\} is a straightforward extension to specify a parsing expression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} (with enclosing braces $\{~{}~{}\}$) that extracts its matched string as a tree node. The extracted node is labeled by the given \#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}, which is used to identify the type of tree nodes.

To start, let us consider a simple example, VAL rule, which recognizes a sequence of numeric characters. Here, Val is a capture version of VAL whose matched strings are constructed as a node labeled as #Int.

Let us start a token extraction.

⬇

    *VAL* = **[0-9]**\+

    *Val* = { **[0-9]**\+ #**Int** }

We use x{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}~{}\xrightarrow{{\color[rgb]{0.3,0,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.3,0,0.5}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}}}~{}~{}}v to write that an expression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} parses an input string $x$ and then transforms it into a tree $v$.

Here, we show how Val parses the string 123:

123 {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}~{}\xrightarrow{{\color[rgb]{0.3,0,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.3,0,0.5}Val}}~{}~{}} #Int123

The nested capture is naturally interpreted as a nested containment of trees that are constructed inside the braces. The following Prod2 parses from 123*45 to a tree depicted in the following manner:

⬇

    *Prod2* = { *Val* **’*’** *Val* #**Mul** }

123*45 {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}~{}\xrightarrow{{\color[rgb]{0.3,0,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.3,0,0.5}Prod2}}~{}~{}} #Mul#Int45#Int123

The Prod2 accepts only a single multiplication expression. Multiple multiplications such as 125 can be captured in two different forms.

⬇

    *ProdM* = { *Val* (**’*’** *Val* )\* #**Mul** }

    *Prod* = { *Val* (**’*’** *Prod* ) #**Mul** } \/ *Val*

The ProdM and Prod both accept the same inputs, while they produce different forms of trees. The ProdM uses a repetition which forms a variable-length list of trees. The Prod, on the other hand, uses a recursion which forms a recursively nested tree.

123456 {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}~{}\xrightarrow{{\color[rgb]{0.3,0,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.3,0,0.5}ProdM}}~{}~{}} #Mul #Int6#Int45#Int123

123456 {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}~{}\xrightarrow{{\color[rgb]{0.3,0,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.3,0,0.5}Prod}}~{}~{}} #Mul #Mul #Int6#Int45#Int123

As shown, we can switch a form of trees by repetition and recursion.

2.2.2. Fold-Capture

In the previous section, we see that a recursion of capturing produces a nested form of recursive trees. It is important to note that the produced trees are always in the right-associative form since PEGs do not allow left recursions.

⬇

    *ProdL* = { (*ProdL* \/ *Val*) **’*’** *Val* #**Mul** }

This example suggests that the capture does not well describe arbitrary forms of syntax trees, especially left-associative binary operators.

CPEG additionally provides the fold-capture annotation, denoted as {\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge}\{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}, which constructs a tree containing the left-hand tree as its first subtree.

Here is an example of a fold-capture version of Prod2, where the first Val is factored outside and then folded inside by {\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge}.

⬇

    *Prod2* = *Val* \^{ **’*’** *Val* #**Mul** }

The {\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge} operator is left-associative. The trees on the left hand are folded one after another.

⬇

    ((*Val* \^{ **’*’** *Val* #**Mul**} ) ..) \^{ **’*’** *Val* #**Mul**}

We use the repetition to denote this iterative folding. Finally, the left-associative version of Prod is described as follows:

⬇

    *ProdL* = *Val* (\^{ **’*’** *Val* #**Mul** })\*

123456 {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}~{}\xrightarrow{{\color[rgb]{0.3,0,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.3,0,0.5}ProdL}}~{}~{}} #Mul #Int6#Mul #Int45#Int123

In fact, left-recursion is a significant restriction of PEG. Although there is a known algorithm for eliminating any left-recursion from a grammar (Parr et al., 2014), this algorithm does not ensure the left-associative. Using fold-capture annotation, we can replace the left-recursion under the impression of keeping the associativity.

2.3. Typing with Regular Expression Types

The declarative tree annotations in Nez are convenient and powerful enough to express many types of syntax trees, ranging from XML and JSON to Java and JavaScript (Kuramitsu, 2016). Trees that the parser users receive are untyped, which are the so-called common trees that are formed in a common structure.

To traverse the trees safely, the users need to check whether the traversed tree is in an intended structure (i.e. its label and arity), as an XML schema variation. Nevertheless, embedding the checking code in a traversal function makes the program fuzzier and prone to errors (Petricek et al., 2016).

Besides, the traversal function should be implemented carefully so that the function is exhaustive—that is, some rule should be applied for all possible structured input trees. Static analysis techniques for checking exhaustiveness checking are based on a type of input value. For instance, XDuce (Hosoya and Pierce, 2003), which is a statically typed XML processing language, provides an exhaustiveness checker that is found on a type called regular expression type (Hosoya et al., 2005). Regular expression types are a variant expression of tree automata (Comon et al., 2007), which has been developed in the contexts of XML schema validations.

As our first attempt at the schematic variation on the trees and the statical exhaustiveness checking, we propose typing rules that infer a regular expression type for a given grammar.

Using regular expression types, all trees that can be parsed from ProdM in the previous section have a type $\mathtt{ProdM}$, which is defined as follows:

[TABLE]

Here, the type {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}[\mathtt{Val},\mathtt{Val}^{\ast}] describes a tree with {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}} that has subtrees typed by the elements of [ ] . The ‘,’, ‘$\ast$’, and $\mathtt{Empty}$ inside respectively denotes concatenation, repetition, and empty tree as regular expression operators. This indicates that the {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}} tree has one or more subtrees that are typed with $\mathtt{Val}$, which is a type variable defined in the second line. The type {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{Empty}] represents no more subtrees and it’s type is Int.

Regular expression types allow recursive type definition. All trees that can be parsed from ProdL are described in the following type:

[TABLE]

3. Formal Definition of CPEG

CPEG is a subset of Nez grammar (Kuramitsu, 2016) to highlight a focused extension to the construction of labeled unranked trees.

3.1. Grammar

A CPEG $G$ is a 5-tuple G=(N_{G},\Sigma,P_{G},{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{s},\mathcal{S}), where $N_{G}$ is a finite set of non-terminal symbols, $\Sigma$ is a finite set of terminal symbols, $P_{G}$ is a finite set of production rules, {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{s} is a start expression and $\mathcal{S}$ is a finite set of label symbols.

Each production rule $r\in P_{G}$ is a pair (A,{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}), that is written as A\leftarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}, where $A\in N_{G}$ and {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} is an expression. For any $A\in N_{G}$, there is exactly one {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} such that A\leftarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}. We regard $P_{G}$ as a function from non-terminals to expressions.

The syntax of an expression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} is defined in Figure 1.

The syntax of CPEGs is reasonably simple. Due to the following syntactic sugars (also defined in (Ford, 2004)), CPEG expresses all the parsing expressions that are constituted by PEG operators (also in Table 1).

[TABLE]

Similarly, the fold-capture annotation can be rewritten by the capture annotation, such as:

[TABLE]

An important exception is the repetitive combination such as {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}(\wedge\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}){\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\ast}, which can not be rewritten by the capture annotation. Thus, we focus only on the repetitive fold-capture as defined in Figure 1. Note that {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}(\wedge\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}){\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\ast} is a syntax sugar of \{{A{\;}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\} where A=\{{A{\;}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}. However, this syntactic sugar contains a left recursion that never terminates in PEGs.

Well-Formed Grammars

As will be discussed in Section 4, a regular expression type is inferred for each CPEG. Some inferred types, however, do not hold a well-formedness condition (Hosoya et al., 2005) without any syntactic restriction on CPEG. Here, we impose an additional restriction to the syntax of CPEG. We refer to the CPEG that holds the restriction as well-formed grammar.

Well-formed grammar is a CPEG that holds either of the following two conditions:

• •

recursive use of nonterminals occurs only in tail positions.

• •

if a sequence A~{}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} exists in the production rules, where $A$ is a recursively used nonterminal and {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} is a some expression (i.e., $A$ is not in tail positions); then {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} has neither capture nor fold-capture as a subexpression.

For instance, the CPEG that has a production rule as the following:

[TABLE]

is not a well-formed grammar, because $A$ is not in the tail position and the subsequent expression of $A$ is a fold-capture.

Whereas, the CPEG that has a production rule as follows:

[TABLE]

is a well-formed grammar, because $A$ is in the tail position.

Furthermore, the CPEG which have a production rule as follows:

[TABLE]

is a well-formed grammar while $A$ is not in the tail position, because the subsequent expression of $A$ is a terminal $a$.

Naturally, the CPEG which have production rules:

[TABLE]

is a well-formed grammar as well.

3.2. Formal Interpretation of CPEG

The semantics is defined as a relation from expressions to trees.

3.2.1. Tree

We start by defining a textual notation to denote a tree for convenience.

The tree of CPEG is a labeled unranked tree on $\mathcal{S}$ and $\Sigma^{\ast}$, where the nodes are labeled with {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\in\mathcal{S} and the leaves hold a string $x\in\Sigma^{\ast}$.

Let $\mathcal{T}_{\mathcal{S}\times\Sigma^{\ast}}$ be a set of trees on $\mathcal{S}$ and $\Sigma^{\ast}$. The syntax of a tree $v\in\mathcal{T}_{\mathcal{S}\times\Sigma^{\ast}}$ is defined as follows.

[TABLE]

Node {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v] denotes the node that is labeled with a symbol {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}} and has subtrees denoted by $v$. We use a concatenation operator , to handle multiple subtrees. Since the order of subtrees is preserved, the concatenation , is not commutative. The notation $x$ denotes a string including empty string. We assume that a concatenation of two strings is equal to a single string. That is $x,x^{\prime}$ is equal to $xx^{\prime}$. Additionally, we assume a concatenation of a node and a string is equal to the node. That is {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v],x and x,{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v] are equal to {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v]. This premise reads that if a tree {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v],x or x,{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v] is constructed, the tree can be regarded as {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v]. In other words, a string that is concatenated with a node can be ignored.

Here, we show the same trees with both pictorial and textual notations as follows.

#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}6#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}45#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}123

{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}[{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{123}],{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{45}],{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{6}]]

#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}6#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}45#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}123

{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}[{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Mul}}[{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{123}],{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{45}]],{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{6}]]

3.2.2. Operational Semantics

To formalize the semantics of a grammar $G$, we define a relation $\Downarrow_{G}$ from ({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e},x) to $(o,y)$, where {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} is an expression of CPEG, $x$ is an input string, $y$ is an unconsumed string, and $o\in\mathcal{T}_{\mathcal{S}\times\Sigma^{\ast}}\cup\{f\}$ is an output. If $o$ is in the form of a tree $v$, the matching succeeds and the tree $v$ is constructed from an input string $x$. If $o=f$, the distinguished symbol $f$ indicates failure.

For (({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e},x),(o,y))\in\Downarrow_{G}, we write {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}\Downarrow_{y}^{x}o. This is read as “an expression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} parses an input $x$ and transforms it to an output $o$ with an unconsumed string $y$”.

Now supposing $v\in\mathcal{T}_{\mathcal{S}\times\Sigma^{\ast}}$, $a,b,c\in\Sigma$, $~{}x,y,z\in\Sigma^{\ast}$, and $\varepsilon$ is an empty string, $\Downarrow^{x}_{y}$ is the smallest relation closed under the set of rules shown in Figure 2.

The matching interpretation in CPEG is the same as PEG (Ford, 2004), except for some tree construction. The capture and fold-capture are an explicit tree constructor as in (E-Capture1) and (E-FoldCap1). All the trees constructed in the subexpressions are contained in a newly constructed tree.

The rules for fold-capture {{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge\kern-3.00003pt{{}^{\ast}}}\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}} are (E-FoldCap1), (E-FoldCap2) and (E-FoldCap3). By (E-FoldCap1), a left-associative tree is derived from {{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge\kern-3.00003pt{{}^{\ast}}}\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}}, if the presuppositions of the rule are satisfied. The presuppositions are summarized in two conditions. The first condition is that the subexpression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1} derives a tree $v_{1}$ from an input string $xy_{1}y_{2}\dots y_{n}z$. Now we denote the unconsumed string by $y_{1}y_{2}\dots y_{n}z$. The second condition is that subexpression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2} derives a tree $v_{2}$ from the string $y_{1}y_{2}\dots y_{n}z$. Additionally, if tree $v_{3}$, tree $v_{4}$, $\cdots$ and tree $v_{n}$ are derived, by applying {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2} repeatedly to the unconsumed string until the out put goes failure, then the derived tree of {{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge\kern-3.00003pt{{}^{\ast}}}\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}} is a left-associative tree:

[TABLE]

The tree $v_{1}$ is contained at the left branch that is $n-1$ times nested from the root. The other tree $v_{i}$ ($2\leq i\leq n$) is stored at the right branch that is $n-i+1$ times nested from the root. The derived tree is also described pictorially as below.

{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}$${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}$${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}$${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}$$v_{1}$$v_{2}$$v_{3}$$v_{n-1}$$v_{n}

By (E-FoldCap2), if the subexpression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1} goes to failure, {{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge\kern-3.00003pt{{}^{\ast}}}\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}} also goes to failure. By (E-FoldCap3), if the subexpression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1} derives $v_{1}$ but the subexpression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2} goes to failure, {{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{1}}{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\wedge\kern-3.00003pt{{}^{\ast}}}\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{2}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\}} derives just $v_{1}$. Note that $v_{1}$ is not nested in the node with {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}.

4. Typing CPEG

4.1. Regular Expression Types

In this section, we review the syntax and the semantics of regular expression types (RETs) (Hosoya et al., 2005).

A RET $\mathtt{T}$ is inductively defined as follows:

[TABLE]

Here, the semantics of RETs is given by the relation $v:\mathtt{T}$ ($v\in\mathcal{T}_{\mathcal{S}\times\Sigma^{\ast}}$), read “the tree $v$ has type $\mathtt{T}$” —the smallest relation closed under the set of typing rules in Figure 3.

The empty sequence $\mathtt{Empty}$, the concatenation $\mathtt{T_{1},T_{2}}$, the union $\mathtt{T_{1}}|\mathtt{T_{2}}$, and the repetition $\mathtt{T^{*}}$ come from regular expressions, as its name implies. The label {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[\mathtt{T}] represents a tree that contains a subtree that has a type $\mathtt{T}$.

The $\mathtt{X}$ represents a type variable that binds an arbitrary RET. The bindings of type variables are given by a single global set $E$ of type definitions of the following form:

[TABLE]

We regard $E$ as a mapping from type variables to their bodies and write $E(\mathtt{X})$ for a reference to the mapped type from X in $E$.

As shown above, the type variables allow recursive types. The readers should note that the expressiveness of RETs correspond to some of the context-free grammars, despite the fact that its name is regular.

4.2. Subtyping

The subtype relation between two types is defined semantically: two types $\mathtt{S},\mathtt{T}$ are in the subtype relation $<:$ if and only if $v:\mathtt{S}$ implies $v:\mathtt{T}$

[TABLE]

As Hosoya (Hosoya et al., 2005)　indicated, the semantic notion of subtype relation over RETs immediately corresponds to the notion of inclusion relation on the set theory.

4.3. Type Inference for CPEG

CPEG intends to infer the types of trees from a grammar before constructing concrete trees.

To start, we consider that a tree $v$ is derived from an input $x$ with a CPEG G=(N_{G},\Sigma,P_{G},{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{s},\mathcal{S}). Since the tree is constructed by the derivation {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{s}\Downarrow^{x}_{y}v, the type of $G$ is regarded as the type of the start expression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}_{s}.

Now, we define type inference rules for CPEG expressions. Let $E$ be a single global set of type bindings and $\Gamma$ be a type environment mapping from non-terminals to type variables. The mappings in $\Gamma$ are denoted by $A:\mathtt{X_{A}}$.

The inference rules are defined as a typing relation denoted by \Gamma\vdash{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}:\mathtt{T}~{}|_{\chi}~{}E, which can be read “under a typing environment $\Gamma$, an expression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} has a type T with a global set $E$”. The relations are the smallest relation closed under the set of typing rules shown in Figure 4.

The $\chi$ is a set of type variables. The $\chi$ is used to store the type variables introduced in each subderivation, and $\mathtt{T}~{}|_{\chi}~{}E$ ensure that the variables appearing in $\mathtt{T}$ are fresh, for each condition of $\chi$ prevent us from building a derivation in which the same variable is used as “fresh” in two different places. Since there is an infinite supply of type variable names, we can always find a way to satisfying the condition. These conventions come from Pierce’s textbook (Pierce, 2002).

The rules (T-Nt1) and (T-Nt2) are types for nonterminal symbols. The premise $A:\mathtt{T}\not\in\Gamma$ in (T-Nt1) is simply an explicit reminder: first, we check if the relation $A:\mathtt{T}$ is included in $\Gamma$; if not, we add that relation to $\Gamma$ and then try to calculate $\mathtt{T}$ for $P_{G}(A)$. In the rule (T-Nt1), the sets $E$ and $\chi$ are updated to $E\cup\{\mathtt{type~{}X_{A}}=\mathtt{T}\}$ and $\chi\cup\{\mathtt{X_{A}}\}$ respectively. The rules (T-Seq), (T-Alt), and (T-FoldCap) update the global set and $\chi$ as well.

Now, let us consider the typing of the following CPEG:

[TABLE]

Note that $[0-9]$ is a derived form of $0/1/2/3/\cdots/7/8/9$.

The derivation of typing is shown in Figure 5.

First, \{{[0-9]}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}\} has the type {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{Empty}] by (T-Capture), where both $E$ and $\chi$ are empty. Although $[0-9]$ in this premise has some type, we omit this derivation for readability. We derive that $Val$ has a type variable $\mathtt{X_{3}}$ by (T-Nt1). At this derivation, the empty global set is updated to $E_{2}$ as shown in the bottom of Figure 5. Moreover, the empty $\chi$ is updated to $\{\mathtt{X_{3}}\}$. Note that $\mathtt{X_{3}}$ is newly introduced in this step.

The next derivation $\APLstar Val$ is a little complex. Hence, we make a sub-derivation tree $\mathcal{D}$. The sub-derivation proceeds similarly from the upper left of the tree to the bottom. As a result of $\mathcal{D}$, the following is derived.

[TABLE]

where, {E_{3}}=\{\mathtt{type~{}X_{4}}={\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{Empty}]\}.

Now, we can go back to the main derivation tree. The following is derived by (T-FoldCap).

[TABLE]

where, {E_{1}}=\{\mathtt{type~{}X_{2}}={\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Prod}}[\mathtt{X_{2}},~{}\mathtt{Empty},\mathtt{X_{4}}]~{}|~{}\mathtt{X_{3}}

,\mathtt{type~{}X_{3}}={\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{Empty}],\mathtt{type~{}X_{4}}={\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{Int}}[\mathtt{Empty}]\}.

Finally, applying (T-Nt1), $\mathtt{X_{1}}$ and the global set $E$:

[TABLE]

are derived.

5. Properties of Typing Rules for CPEG

In this section, we show type soundness and uniqueness of types property of the typing rules.

First, we have shown the uniqueness of types property, which means that the inferred type is always unique for a given CPEG. Furthermore, this property says that the derivation tree is also deterministic.

Theorem 5.1 (Uniqueness of Types).

In a given typing context $\Gamma$, an expression {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e} has uniquely one type with type variables all in the domain of a global set of types $E$. Moreover, there is just one derivation of this typing built from the typing rule that derives the typing relation.

Proof.

The proof goes by the structural induction on {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}. ∎

Next, we have shown type soundness. Intuitively, type soundness states that if a CPEG has type $\mathtt{T}$ under the type inference, then the trees derived from the CPEG are typed by the type $\mathtt{T}$.

Theorem 5.2 (Soundness).

Let $G$ be a CPEG such that $G=(N_{G},\Sigma,$ $P_{G},e_{s},\mathcal{S})$. Let $E$ be a global set of regular expression types.

[TABLE]

Proof.

By induction on derivation of {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}\Downarrow^{x}_{y}v. The (E-Empty), (E-Term1), (E-Not2) cases are immediate by the rules (T-Empty), (T-Term), (T-Not), (S-Empty). For the other cases, we will discuss the (E-Capture1) case and the proof for the remaining rules proceed in the same manner.

Case E-Capture1: The start expression is \{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\} and the derived tree is {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v] such that {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}\Downarrow_{y}^{x}v. Let the following be an induction hypothesis.

[TABLE]

The only typing rule for \{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}e}}\;\#{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}\} is the rule (T-Capture). By inversion of (T-Capture),

[TABLE]

Using the induction hypothesis, $v$ has type $\mathtt{T}$. Finally, {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[v]:{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\mathbf{L}}[\mathtt{T}] by (S-Node). ∎

6. Related Work

Here we review parsing with an emphasis on tree construction technique.

PEGs have been gaining popularity among many language developers since it was presented by B. Ford in 2004. In the context of PEGs, a parser generator is a standard approach to the development of parsers (Grimm, 2006). PEGs, as well as other CFG-based grammars, can well describe the syntactic pattern of the input while providing a very poor specification of the output. Consequently, the construction of syntax trees is mostly left to the embedded action code, which is usually written in a target programming language. However, the use of action code reduces a good property of declarative grammar specification. Moreover, their additional code generation and compilation process are cumbersome compared to regex-style matching and capturing.

Parser combinators such as Parsec (Leijen and Meijer, 2002), FParsec (Tolksdorf, 2013), and Scala Parser Combinators (Moors et al., 2008) provide a more integrated means for writing a recursive descent parser. Although parser combinators are not always based on some grammar formalism, many PEG parsers have been implemented by the combinators. In parser combinators, the resulting syntax trees can be well-typed. The tree construction, however, relies on code fragments that directly manipulates parsing results. Consequently, the specification of a parser is still hard to maintain (Klint et al., 2010; Adams, 2013).

More recently, declarative parsing (no action code) has been focused in many grammar formalisms, since action code makes it difficult to maintain a parser specification and reduces grammar reusability (Klint et al., 2010; Adams, 2013). In the contexts of PEGs, LPeg (Ierusalimschy, 2009) is implemented as a PEG-based pattern matching tool that provides a grouped capture like regex. Moreover, Nez grammar (Kuramitsu, 2016, 2015), an ancestor of CPEG, provides a structured capture that can construct complex syntax trees. These parse data require no action code, but they are untyped.

For the parser users, types are significant. This viewpoint has made another attempt to parser generations from data types. Notably, PADS/ML (Mandelbaum et al., 2007) describes DDCα (Mandelbaum et al., 2007; Fisher et al., 2006) and then generates a parser from the data specification. The “parser from types” approach can follow various syntax patterns including programming languages. However, formally specifying the provided types safety would be more challenging (Petricek et al., 2016).

Finally, a type system for grammar is new. As a starting point, we use RETs, which have intensively been studied in the context of XML schemas and tree automata (Hosoya and Pierce, 2003; Hosoya et al., 2005). We consider that RETs are a straightforward type representation of syntax trees, and RETs can make a theoretical bridge between declarative parsing, tree automata, and programming language design. Indeed, binding RETs with ML and OCaml has been reported in (Sulzmann and Lu, 2006; Frisch, 2006).

7. Conclusion

Regular expressions, or regexes, have had great success both as pattern matching and as a library tool to develop small parsers. However, the absence of recursive patterns results in very limited parser applications. Since PEGs are more powerful than regular expressions, PEGs with regex-like captures could make it much easier to integrate a full-fledged parser into programs.

CPEGs are a formally developed extension of PEGs with regex-like captures. Two annotations (capture and fold-capture) allow a flexible construction of complex syntax trees. More importantly, a CPEG tree is a company with regular expression types that are a foundation of XML schema and tree automata. A regular expression type for a given CPEG is inferred syntactically. We present a formal definition of the type inference and we proved its soundness and uniqueness of types property.

Our attempt to type system for grammar is new. There are several interesting issues that remain unexplored. The future direction is that we will implement the CPEG based parser and investigate a practical aspect of CPEG. We intend to implement the CPEG based parser in F# and integrate our type inference into F# type provider (Syme et al., 2012).