Towards Interpretable Summary Evaluation via Allocation of Contextual Embeddings to Reference Text Topics

TL;DR

This paper introduces MISEM, a new interpretable method for evaluating summaries by allocating contextual embeddings to reference text topics, enhancing transparency and qualitative analysis.

Contribution

The paper presents MISEM, a novel evaluation approach that assigns summary embeddings to reference topics, along with an interpretability toolbox for automated and interactive analysis.

Findings

MISEM achieves a .404 Pearson correlation with human judgment.

The interpretability toolbox enables transparent assessment of summaries.

Allocation of embeddings improves qualitative understanding of summary quality.

Abstract

Despite extensive recent advances in summary generation models, evaluation of auto-generated summaries still widely relies on single-score systems insufficient for transparent assessment and in-depth qualitative analysis. Towards bridging this gap, we propose the multifaceted interpretable summary evaluation method (MISEM), which is based on allocation of a summary's contextual token embeddings to semantic topics identified in the reference text. We further contribute an interpretability toolbox for automated summary evaluation and interactive visual analysis of summary scoring, topic identification, and token-topic allocation. MISEM achieves a promising .404 Pearson correlation with human judgment on the TAC'08 dataset.