Scalable Econometrics on Big Data -- The Logistic Regression on Spark

TL;DR

This paper addresses the challenge of performing logistic regression on large datasets using Spark, introducing a new PySpark package that provides statistical summaries for inference, thus enhancing econometric analysis at scale.

Contribution

The paper develops a PySpark package that efficiently computes statistical summaries for logistic regression, improving econometric analysis on big data.

Findings

The package successfully computes statistical summaries for large-scale logistic regression.

It demonstrates robustness and efficiency in handling big datasets with Spark.

Facilitates statistical inference in big data econometrics.

Abstract

Extra-large datasets are becoming increasingly accessible, and computing tools designed to handle huge amount of data efficiently are democratizing rapidly. However, conventional statistical and econometric tools are still lacking fluency when dealing with such large datasets. This paper dives into econometrics on big datasets, specifically focusing on the logistic regression on Spark. We review the robustness of the functions available in Spark to fit logistic regression and introduce a package that we developed in PySpark which returns the statistical summary of the logistic regression, necessary for statistical inference.