Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?

TL;DR

This paper explores how large language models can be controlled to generate diverse scientific summaries, showing they outperform humans in some tasks but face challenges with long and highly abstractive summaries.

Contribution

The study demonstrates controllability of LLMs for scientific summarization using stylistic controls and keyword guidance, without extensive fine-tuning.

Findings

Non-fine-tuned LLMs outperform humans in MuP review similarity and preferences.

Keyword-based classifier-free guidance improves controllability and lexical overlap.

LLMs struggle with generating summaries longer than 8 sentences and highly abstractive lay summaries.

Abstract

In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.