Large-scale text processing pipeline with Apache Spark

TL;DR

This paper presents a scalable text processing pipeline using Apache Spark to analyze policy diffusion across US state legislatures, overcoming computational challenges of all-pairs comparison in large datasets.

Contribution

It introduces a distributed text processing workflow with Spark and Scala, enabling large-scale policy analysis previously limited by computational constraints.

Findings

Implemented a scalable Spark-based pipeline for policy diffusion analysis

Addressed challenges in unstructured data processing and graph analysis at scale

Demonstrated feasibility of all-pairs comparison in large legislative datasets

Abstract

In this paper, we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas. We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark dataframes and Scala application programming interface. We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.