Grappling with the Scale of Born-Digital Government Publications: Toward Pipelines for Processing and Searching Millions of PDFs

TL;DR

This paper addresses the challenge of processing and searching millions of born-digital government PDFs by proposing scalable pipelines that leverage metadata, textual, and visual features for efficient analysis and discovery.

Contribution

It introduces initial scalable approaches for searching and analyzing large collections of government PDFs using machine learning on textual and visual features.

Findings

Demonstrates the utility of PDF metadata for large-scale analysis

Proposes machine learning methods for search and discovery in PDFs

Outlines operationalization strategies for handling millions of PDFs

Abstract

Official government publications are key sources for understanding the history of societies. Web publishing has fundamentally changed the scale and processes by which governments produce and disseminate information. Significantly, a range of web archiving programs have captured massive troves of government publications. For example, hundreds of millions of unique U.S. Government documents posted to the web in PDF form have been archived by libraries to date. Yet, these PDFs remain largely unutilized and understudied in part due to the challenges surrounding the development of scalable pipelines for searching and analyzing them. This paper utilizes a Library of Congress dataset of 1,000 government PDFs in order to offer initial approaches for searching and analyzing these PDFs at scale. In addition to demonstrating the utility of PDF metadata, this paper offers computationally-efficient machine learning approaches to search and discovery that utilize the PDFs' textual and visual features as well. We conclude by detailing how these methods can be operationalized at scale in order to support systems for navigating millions of PDFs.