CLSE: Corpus of Linguistically Significant Entities

TL;DR

This paper introduces CLSE, a linguistically annotated corpus of named entities across 34 languages, to improve the linguistic accuracy of natural language generation systems, especially for complex languages.

Contribution

The paper presents CLSE, a new multilingual corpus of linguistically significant entities, and demonstrates its use in creating a more linguistically representative NLG evaluation benchmark.

Findings

Established quality baselines for neural, template-based, and hybrid NLG systems.

Created a multilingual NLG benchmark in French, Marathi, and Russian.

Highlighted strengths and weaknesses of different NLG approaches.

Abstract

One of the biggest challenges of natural language generation (NLG) is the proper handling of named entities. Named entities are a common source of grammar mistakes such as wrong prepositions, wrong article handling, or incorrect entity inflection. Without factoring linguistic representation, such errors are often underrepresented when evaluating on a small set of arbitrarily picked argument values, or when translating a dataset from a linguistically simpler language, like English, to a linguistically complex language, like Russian. However, for some applications, broadly precise grammatical correctness is critical -- native speakers may find entity-related grammar errors silly, jarring, or even offensive. To enable the creation of more linguistically diverse NLG datasets, we release a Corpus of Linguistically Significant Entities (CLSE) annotated by linguist experts. The corpus includes 34 languages and covers 74 different semantic types to support various applications from airline ticketing to video games. To demonstrate one possible use of CLSE, we produce an augmented version of the Schema-Guided Dialog Dataset, SGD-CLSE. Using the CLSE's entities and a small number of human translations, we create a linguistically representative NLG evaluation benchmark in three languages: French (high-resource), Marathi (low-resource), and Russian (highly inflected language). We establish quality baselines for neural, template-based, and hybrid NLG systems and discuss the strengths and weaknesses of each approach.