How to Evaluate Controlled Natural Languages

TL;DR

This paper introduces a framework for evaluating controlled natural languages through user experiments, demonstrated with Attempto Controlled English (ACE), to assess their effectiveness in understanding and classification tasks.

Contribution

It proposes a novel evaluation framework for controlled natural languages based on user classification tasks using ontographs, applied to ACE for validation.

Findings

Framework successfully applied to ACE

Users can classify statements with high accuracy

Provides a basis for comparing different controlled languages

Abstract

This paper presents a general framework how controlled natural languages can be evaluated and compared on the basis of user experiments. The subjects are asked to classify given statements (in the language to be tested) as either true or false with respect to a certain situation that is shown in a graphical notation called "ontographs". A first experiment has been conducted that applies this framework to the language Attempto Controlled English (ACE).