Towards objectively evaluating the quality of generated medical summaries

TL;DR

This paper introduces an objective evaluation method for generated medical summaries by counting facts and calculating standard metrics, aiming to improve reproducibility and accuracy in assessment.

Contribution

It presents a novel fact-counting evaluation approach for medical summaries, enhancing objectivity and reproducibility over traditional subjective methods.

Findings

The method provides consistent and objective quality assessments.

It is particularly effective for medical report summarization.

The approach simplifies evaluation processes in medical NLP tasks.

Abstract

We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.