Not too long do read: Evaluating LLM-generated extreme scientific summaries

TL;DR

This paper introduces BiomedTLDR, a new dataset of researcher-authored scientific summaries, and evaluates how well large language models generate scientific TLDRs, revealing their tendencies towards extractiveness over abstraction.

Contribution

The paper presents BiomedTLDR, a novel high-quality dataset for scientific summarization, and provides an empirical evaluation of LLMs' performance in generating scientific TLDRs.

Findings

LLMs tend to produce more extractive summaries than human experts.

Some LLMs can generate humanoid-like summaries, but often rely on lexical and rhetorical similarities.

The dataset enables better evaluation and development of scientific summarization models.

Abstract

High-quality scientific extreme summary (TLDR) facilitates effective science communication. How do large language models (LLMs) perform in generating them? How are LLM-generated summaries different from those written by human experts? However, the lack of a comprehensive, high-quality scientific TLDR dataset hinders both the development and evaluation of LLMs' summarization ability. To address these, we propose a novel dataset, BiomedTLDR, containing a large sample of researcher-authored summaries from scientific papers, which leverages the common practice of including authors' comments alongside bibliography items. We then test popular open-weight LLMs for generating TLDRs based on abstracts. Our analysis reveals that, although some of them successfully produce humanoid summaries, LLMs generally exhibit a greater affinity for the original text's lexical choices and rhetorical structures, hence tend to be more extractive rather than abstractive in general, compared to humans. Our code and datasets are available at https://github.com/netknowledge/LLM_summarization (Lyu and Ke, 2025).