PCoA: A New Benchmark for Medical Aspect-Based Summarization With Phrase-Level Context Attribution

TL;DR

PCoA introduces a new benchmark dataset and evaluation framework for medical aspect-based summarization, emphasizing precise phrase-level context attribution to improve verification and quality of system-generated summaries in high-stakes medical domains.

Contribution

The paper presents PCoA, a novel expert-annotated benchmark with a decoupled evaluation framework for phrase-level context attribution in medical summarization, and validates its effectiveness through extensive experiments.

Findings

PCoA provides a reliable benchmark for medical summarization evaluation.

Explicit identification of relevant sentences and phrases enhances summary quality.

Large language models show improved performance when using phrase-level context attribution.

Abstract

Verifying system-generated summaries remains challenging, as effective verification requires precise attribution to the source context, which is especially crucial in high-stakes medical domains. To address this challenge, we introduce PCoA, an expert-annotated benchmark for medical aspect-based summarization with phrase-level context attribution. PCoA aligns each aspect-based summary with its supporting contextual sentences and contributory phrases within them. We further propose a fine-grained, decoupled evaluation framework that independently assesses the quality of generated summaries, citations, and contributory phrases. Through extensive experiments, we validate the quality and consistency of the PCoA dataset and benchmark several large language models on the proposed task. Experimental results demonstrate that PCoA provides a reliable benchmark for evaluating system-generated summaries with phrase-level context attribution. Furthermore, comparative experiments show that explicitly identifying relevant sentences and contributory phrases before summarization can improve overall quality. The data and code are available at https://github.com/chubohao/PCoA.