Semantic Structural Evaluation for Text Simplification

TL;DR

This paper introduces SAMSA, a novel reference-less measure for evaluating the structural quality of text simplification by leveraging semantic parsing, which correlates well with human judgments.

Contribution

The paper presents the first measure to evaluate structural aspects of text simplification using semantic parsing, addressing limitations of existing lexical-focused metrics.

Findings

SAMSA correlates strongly with human judgments

Existing reference-based measures are inadequate for structural evaluation

SAMSA provides a reference-less, automatic evaluation method

Abstract

Current measures for evaluating text simplification systems focus on evaluating lexical text aspects, neglecting its structural aspects. In this paper we propose the first measure to address structural aspects of text simplification, called SAMSA. It leverages recent advances in semantic parsing to assess simplification quality by decomposing the input based on its semantic structure and comparing it to the output. SAMSA provides a reference-less automatic evaluation procedure, avoiding the problems that reference-based methods face due to the vast space of valid simplifications for a given sentence. Our human evaluation experiments show both SAMSA's substantial correlation with human judgments, as well as the deficiency of existing reference-based measures in evaluating structural simplification.