Automating Historical Insight Extraction from Large-Scale Newspaper Archives via Neural Topic Modeling

TL;DR

This paper demonstrates how neural topic modeling with BERTopic can effectively analyze large-scale historical newspaper archives, revealing evolving themes and discourse patterns over time, especially related to nuclear topics.

Contribution

It introduces the application of BERTopic to historical newspaper data, showing its advantages over traditional methods in capturing complex, evolving discourse.

Findings

BERTopic effectively captures topic evolution over decades.

The method uncovers co-occurrence patterns in nuclear discourse.

It demonstrates scalability and contextual sensitivity in historical analysis.

Abstract

Extracting coherent and human-understandable themes from large collections of unstructured historical newspaper archives presents significant challenges due to topic evolution, Optical Character Recognition (OCR) noise, and the sheer volume of text. Traditional topic-modeling methods, such as Latent Dirichlet Allocation (LDA), often fall short in capturing the complexity and dynamic nature of discourse in historical texts. To address these limitations, we employ BERTopic. This neural topic-modeling approach leverages transformerbased embeddings to extract and classify topics, which, despite its growing popularity, still remains underused in historical research. Our study focuses on articles published between 1955 and 2018, specifically examining discourse on nuclear power and nuclear safety. We analyze various topic distributions across the corpus and trace their temporal evolution to uncover long-term trends and shifts in public discourse. This enables us to more accurately explore patterns in public discourse, including the co-occurrence of themes related to nuclear power and nuclear weapons and their shifts in topic importance over time. Our study demonstrates the scalability and contextual sensitivity of BERTopic as an alternative to traditional approaches, offering richer insights into historical discourses extracted from newspaper archives. These findings contribute to historical, nuclear, and social-science research while reflecting on current limitations and proposing potential directions for future work.