Leveraging Hierarchical Organization for Medical Multi-document Summarization

TL;DR

This study explores how hierarchical organization in multi-document summarization enhances the clarity, coherence, and human preference for medical summaries, outperforming traditional flat methods.

Contribution

It introduces hierarchical structures into large language models for medical MDS and demonstrates their effectiveness through comprehensive evaluations.

Findings

Hierarchical approaches improve summary clarity and coherence.

Human experts prefer hierarchical model summaries over human-written ones.

Hierarchical methods maintain factuality and coverage effectively.

Abstract

Medical multi-document summarization (MDS) is a complex task that requires effectively managing cross-document relationships. This paper investigates whether incorporating hierarchical structures in the inputs of MDS can improve a model's ability to organize and contextualize information across documents compared to traditional flat summarization methods. We investigate two ways of incorporating hierarchical organization across three large language models (LLMs), and conduct comprehensive evaluations of the resulting summaries using automated metrics, model-based metrics, and domain expert evaluation of preference, understandability, clarity, complexity, relevance, coverage, factuality, and coherence. Our results show that human experts prefer model-generated summaries over human-written summaries. Hierarchical approaches generally preserve factuality, coverage, and coherence of information, while also increasing human preference for summaries. Additionally, we examine whether simulated judgments from GPT-4 align with human judgments, finding higher agreement along more objective evaluation facets. Our findings demonstrate that hierarchical structures can improve the clarity of medical summaries generated by models while maintaining content coverage, providing a practical way to improve human preference for generated summaries.