Corpus Annotation for Parser Evaluation

TL;DR

This paper introduces a new corpus annotation scheme for parser evaluation that encodes grammatical relations, applied to a public corpus, and demonstrates its effectiveness in assessing parser accuracy.

Contribution

It presents a novel annotation scheme for parser evaluation and applies it to a new public corpus of English text.

Findings

The annotation scheme effectively encodes grammatical relations.

The corpus can be used to evaluate parser accuracy.

The scheme improves upon existing evaluation methods.

Abstract

We describe a recently developed corpus annotation scheme for evaluating parsers that avoids shortcomings of current methods. The scheme encodes grammatical relations between heads and dependents, and has been used to mark up a new public-domain corpus of naturally occurring English text. We show how the corpus can be used to evaluate the accuracy of a robust parser, and relate the corpus to extant resources.