Adding more data does not always help: A study in medical conversation summarization with PEGASUS

TL;DR

This study investigates how dataset size impacts medical conversation summarization with PEGASUS and finds that performance plateaus, with active learning strategies offering no significant advantage over simple dataset expansion.

Contribution

It provides insights into the effects of dataset size and active learning strategies on medical summarization using PEGASUS, highlighting limitations in low-data regimes.

Findings

Model performance saturates as dataset size increases.

Active learning strategies perform similarly to simple dataset size increase.

Naive pseudo-labeling does not improve and may slightly worsen results.

Abstract

Medical conversation summarization is integral in capturing information gathered during interactions between patients and physicians. Summarized conversations are used to facilitate patient hand-offs between physicians, and as part of providing care in the future. Summaries, however, can be time-consuming to produce and require domain expertise. Modern pre-trained NLP models such as PEGASUS have emerged as capable alternatives to human summarization, reaching state-of-the-art performance on many summarization benchmarks. However, many downstream tasks still require at least moderately sized datasets to achieve satisfactory performance. In this work we (1) explore the effect of dataset size on transfer learning medical conversation summarization using PEGASUS and (2) evaluate various iterative labeling strategies in the low-data regime, following their success in the classification setting. We find that model performance saturates with increase in dataset size and that the various active-learning strategies evaluated all show equivalent performance consistent with simple dataset size increase. We also find that naive iterative pseudo-labeling is on-par or slightly worse than no pseudo-labeling. Our work sheds light on the successes and challenges of translating low-data regime techniques in classification to medical conversation summarization and helps guides future work in this space. Relevant code available at \url{https://github.com/curai/curai-research/tree/main/medical-summarization-ML4H-2021}.