Towards Human-Free Automatic Quality Evaluation of German Summarization

TL;DR

This paper explores adapting the BLANC automatic evaluation metric for German summarization, demonstrating its effectiveness in assessing informativeness without relying on human-generated gold standards.

Contribution

It presents a method to adjust the BLANC metric for German, enabling human-free evaluation of summarization quality in this language.

Findings

BLANC correlates well with human ratings in German.

BLANC effectively evaluates informativeness of summaries.

The adapted BLANC outperforms some existing automatic metrics.

Abstract

Evaluating large summarization corpora using humans has proven to be expensive from both the organizational and the financial perspective. Therefore, many automatic evaluation metrics have been developed to measure the summarization quality in a fast and reproducible way. However, most of the metrics still rely on humans and need gold standard summaries generated by linguistic experts. Since BLANC does not require golden summaries and supposedly can use any underlying language model, we consider its application to the evaluation of summarization in German. This work demonstrates how to adjust the BLANC metric to a language other than English. We compare BLANC scores with the crowd and expert ratings, as well as with commonly used automatic metrics on a German summarization data set. Our results show that BLANC in German is especially good in evaluating informativeness.