SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era

TL;DR

SciZoom is a comprehensive benchmark dataset of nearly 45,000 scientific papers designed to evaluate hierarchical summarization methods and analyze linguistic shifts in scientific writing before and after the widespread adoption of LLMs like ChatGPT.

Contribution

It introduces a large-scale, multi-granularity scientific summarization benchmark across the LLM era, enabling research on summarization and linguistic evolution in scientific discourse.

Findings

Detected significant shifts in phrase patterns and rhetorical styles post-LLM adoption.

Provided evidence of more confident but homogenized scientific writing with LLM assistance.

Created a publicly available dataset for future research in scientific summarization and discourse analysis.

Abstract

The explosive growth of AI research has created unprecedented information overload, increasing the demand for scientific summarization at multiple levels of granularity beyond traditional abstracts. While LLMs are increasingly adopted for summarization, existing benchmarks remain limited in scale, target only a single granularity, and predate the LLM era. Moreover, since the release of ChatGPT in November 2022, researchers have rapidly adopted LLMs for drafting manuscripts themselves, fundamentally transforming scientific writing, yet no resource exists to analyze how this writing has evolved. To bridge these gaps, we introduce SciZoom, a benchmark comprising 44,946 papers from four top-tier ML venues (NeurIPS, ICLR, ICML, EMNLP) spanning 2020 to 2025, explicitly stratified into Pre-LLM and Post-LLM eras. SciZoom provides three hierarchical summarization targets (Abstract, Contributions, and TL;DR) achieving compression ratios up to 600:1, enabling both multi-granularity summarization research and temporal mining of scientific writing patterns. Our linguistic analysis reveals striking shifts in phrase patterns (up to 10x for formulaic expressions) and rhetorical style (23% decline in hedging), suggesting that LLM-assisted writing produces more confident yet homogenized prose. SciZoom serves as both a challenging benchmark and a unique resource for mining the evolution of scientific discourse in the generative AI era. Our code and dataset are publicly available on GitHub (https://github.com/janghana/SciZoom) and Hugging Face (https://huggingface.co/datasets/hanjang/SciZoom), respectively.