A Statistical Framework for Detecting Emergent Narratives in Longitudinal Text Corpora

TL;DR

This paper introduces a statistical framework using LDA to detect emerging narratives in long-term text data, exemplified by economic publications, by identifying sustained increases in topic prominence.

Contribution

It presents a novel method for identifying narrative emergence through modeling topic trajectories as latent variables over time.

Findings

Topics related to Nobel-winning contributions show sustained prevalence increases.

The framework correlates topic prominence with citation activity and recognition.

It provides a statistically grounded approach for thematic change analysis.

Abstract

Narratives about economic events and policies are widely recognised as influential drivers of economic and business behaviour. Yet the statistical identification of narrative emergence remains underdeveloped. Narratives evolve gradually, exhibit subtle shifts in content, and may exert influence disproportionate to their observable frequency, making it difficult to determine when observed changes reflect genuine structural shifts rather than routine variation in language use. We propose a statistical framework for detecting narrative emergence in longitudinal text corpora using Latent Dirichlet Allocation (LDA). We define emergence as a sustained increase in a topic's relative prominence over time and articulate a statistical framework for interpreting such trajectories, recognising that topic proportions are latent, model-estimated quantities. We illustrate the approach using a corpus of academic publications in economics spanning 1970-2018, where Nobel Prize-recognised contributions serve as externally observable signals of influential narratives. Topics associated with these contributions display sustained increases in estimated prevalence that coincide with periods of heightened citation activity and broader disciplinary recognition. These findings indicate that model-based topic trajectories can reflect identifiable shifts in economic discourse and provide a statistically grounded basis for analysing thematic change in longitudinal textual data.