GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models

TL;DR

This paper investigates the prevalence of fabricated citations by Large Language Models and their impact on academic trust, providing a large-scale analysis and a framework for citation verification.

Contribution

It introduces exttt{ extbackslash citeb}, a framework for large-scale citation verification, and presents a comprehensive study on citation validity in the era of LLMs.

Findings

All tested LLMs hallucinate citations at rates from 14.23% to 94.93%.

1.07% of analyzed papers contain invalid citations, increasing by 80.9% in 2025.

Most researchers and reviewers do not thoroughly verify citations, risking academic integrity.

Abstract

Citations provide the basis for trusting scientific claims; when they are invalid or fabricated, this trust collapses. With the advent of Large Language Models (LLMs), this risk has intensified: LLMs are increasingly used for academic writing, but their tendency to fabricate citations (``ghost citations'') poses a systemic threat to citation validity. To quantify this threat, we develop \citeb, an open-source framework for large-scale citation verification, and conduct a comprehensive study of citation validity in the LLM era through three complementary experiments. First, we benchmark 13 LLMs on citation generation task in various research domains, finding that all models hallucinate citations at rate from 14.23\% to 94.93\%. Second, we analyze 2.2 million citations from 56,381 papers at AI/ML and Security venues (2020--2025), finding that 1.07\% of papers contain invalid citations, with an 80.9\% increase in 2025. Third, we survey 97 researchers, finding that 87.2\% use AI-powered tools in their workflows, 76.7\% of reviewers do not thoroughly check references, and 74.5\% view peer review as ineffective at catching citation errors. Based on these findings, we argue that ghost citations represent a systemic threat to academic integrity, and call for coordinated efforts from community to address this challenge.