Goodness-of-Fit Tests for Large Datasets

TL;DR

This paper introduces a scalable, model-free goodness-of-fit testing method for large datasets, based on the empirical distribution and suitable for distributed computing environments like Spark.

Contribution

It presents a parallelizable goodness-of-fit test for Big Data that does not rely on model assumptions, integrating inferential statistics with large-scale data analysis.

Findings

Method is applicable to distributed datasets

Based on empirical distribution without model assumptions

Easily parallelizable for cluster computing

Abstract

Nowadays, data analysis in the world of Big Data is connected typically to data mining, descriptive or exploratory statistics, e.~g.\ cluster analysis, classification or regression analysis. Aside these techniques there is a huge area of methods from inferential statistics that are rarely considered in connection with Big Data. Nevertheless, inferential methods are also of use for Big Data analysis, especially for quantifying uncertainty. The article at hand will provide some insights to methodological and technical issues referring inferential methods in the Big Data area in order to bring together Big Data and inferential statistics, as it comes along with its difficulties. We present an approach that allows testing goodness-of-fit without model assumptions and relying on the empirical distribution. Especially, the method is able to utilize information from large datasets. Thereby, the approach is based on a clear theoretical background. We concentrate on the widely-used Kolmogorov-Smirnov test that is applied for testing goodness-of-fit in statistics. Our approach can be parallelized easily, which makes it applicable to distributed datasets particularly on a compute cluster. By this contribution, we turn to an audience that is interested in the technical and methodological backgrounds while implementing especially inferential statistical methods with Big Data tools as e. g. Spark.