Structural shifts in institutional participation and collaboration within the AI arXiv preprint research ecosystem

TL;DR

This study analyzes how the AI research landscape on arXiv has evolved from 2021 to 2025, highlighting a surge in publications post-ChatGPT and persistent institutional collaboration gaps.

Contribution

It introduces a multi-stage data pipeline and LLM-based classification to examine structural shifts in AI research collaboration and publication patterns.

Findings

Surge in AI publications after ChatGPT's release

Academic institutions dominate research output

Collaboration between academia and industry remains limited

Abstract

The emergence of large language models (LLMs) represents a significant technological shift within the scientific ecosystem, particularly within the field of artificial intelligence (AI). This paper examines structural changes in the AI research landscape using a dataset of arXiv preprints (cs.AI) from 2021 through 2025. Given the rapid pace of AI development, the preprint ecosystem has become a critical barometer for real-time scientific shifts, often preceding formal peer-reviewed publication by months or years. By employing a multi-stage data collection and enrichment pipeline in conjunction with LLM-based institution classification, we analyze the evolution of publication volumes, author team sizes, and academic--industry collaboration patterns. Our results reveal an unprecedented surge in publication output following the introduction of ChatGPT, with academic institutions continuing to provide the largest volume of research. However, we observe that academic--industry collaboration is still suppressed, as measured by a Normalized Collaboration Index (NCI) that remains significantly below the random-mixing baseline across all major subfields. These findings highlight a continuing institutional divide and suggest that the capital-intensive nature of generative AI research may be reshaping the boundaries of scientific collaboration.