jp-evalb: Robust Alignment-based PARSEVAL Measures

TL;DR

jp-evalb is a new alignment-based evaluation system for constituency parsing that addresses limitations of the traditional evalb method, providing more flexible and accurate performance assessments.

Contribution

It introduces a novel alignment-based framework, jp-evalb, improving constituency parsing evaluation by overcoming evalb's dependency on tokenization and sentence boundary consistency.

Findings

jp-evalb offers more accurate parsing evaluation.

It handles tokenization discrepancies better.

The method is more adaptable across different datasets.

Abstract

We introduce an evaluation system designed to compute PARSEVAL measures, offering a viable alternative to \texttt{evalb} commonly used for constituency parsing evaluation. The widely used \texttt{evalb} script has traditionally been employed for evaluating the accuracy of constituency parsing results, albeit with the requirement for consistent tokenization and sentence boundaries. In contrast, our approach, named \texttt{jp-evalb}, is founded on an alignment method. This method aligns sentences and words when discrepancies arise. It aims to overcome several known issues associated with \texttt{evalb} by utilizing the `jointly preprocessed (JP)' alignment-based method. We introduce a more flexible and adaptive framework, ultimately contributing to a more accurate assessment of constituency parsing performance.