Keyphrase Based Evaluation of Automatic Text Summarization

TL;DR

This paper introduces KpEval, a keyphrase-based evaluation method for automatic text summarization that improves correlation with existing systems by focusing on important concepts conveyed through keyphrases.

Contribution

The study presents a novel keyphrase-based evaluation system that enhances summary assessment accuracy by emphasizing core concepts and demonstrates strong correlation with established evaluation metrics.

Findings

KpEval correlates highly with AutoSummENG MeMoG (Pearson 0.8840, Spearman 0.9667).

KpEval outperforms traditional n-gram matching methods in summary evaluation.

The approach is effective on Arabic multi-document datasets.

Abstract

The development of methods to deal with the informative contents of the text units in the matching process is a major challenge in automatic summary evaluation systems that use fixed n-gram matching. The limitation causes inaccurate matching between units in a peer and reference summaries. The present study introduces a new Keyphrase based Summary Evaluator KpEval for evaluating automatic summaries. The KpEval relies on the keyphrases since they convey the most important concepts of a text. In the evaluation process, the keyphrases are used in their lemma form as the matching text unit. The system was applied to evaluate different summaries of Arabic multi-document data set presented at TAC2011. The results showed that the new evaluation technique correlates well with the known evaluation systems: Rouge1, Rouge2, RougeSU4, and AutoSummENG MeMoG. KpEval has the strongest correlation with AutoSummENG MeMoG, Pearson and spearman correlation coefficient measures are 0.8840, 0.9667 respectively.