AstroMLab 5: Structured Summaries and Concept Extraction for 400,000 Astrophysics Papers

TL;DR

This paper introduces a large dataset of astrophysics papers with structured summaries and concept extraction, enabling advanced semantic analysis and discovery in the field.

Contribution

It provides a comprehensive dataset with structured summaries and detailed concept embeddings for 400,000 astrophysics papers, enhancing semantic understanding and retrieval.

Findings

Concepts offer denser coverage than traditional keywords.

Semantic embeddings enable diverse discovery pathways.

The dataset supports advanced analysis of astrophysics literature.

Abstract

We present a dataset of 408,590 astrophysics papers from arXiv (astro-ph), spanning 1992 through July 2025. Each paper has been processed through a multi-stage pipeline to produce: (1) structured summaries organized into six semantic sections (Background, Motivation, Methodology, Results, Interpretation, Implication), and (2) concept extraction yielding 9,999 unique concepts with detailed descriptions. The dataset contains 3.8 million paper-concept associations and includes semantic embeddings for all concepts. Comparison with traditional ADS keywords reveals that the concepts provide denser coverage and more uniform distribution, while analysis of embedding space structure demonstrates that concepts are semantically dispersed within papers-enabling discovery through multiple diverse entry points. Concept vocabulary and embeddings are publicly released at https://github.com/tingyuansen/astro-ph_knowledge_graph.