Using a Power Law Distribution to describe Big Data

TL;DR

The paper proposes a method to model big data with a power law distribution, enabling efficient filtering of uninteresting data and improving data analysis scalability.

Contribution

It introduces a technique to derive a power law background model from arbitrary datasets, aiding in data filtering and scalability for big data applications.

Findings

Model accurately fits social sensor data distributions

Enables automatic identification of high degree nodes

Scales effectively with large datasets

Abstract

The gap between data production and user ability to access, compute and produce meaningful results calls for tools that address the challenges associated with big data volume, velocity and variety. One of the key hurdles is the inability to methodically remove expected or uninteresting elements from large data sets. This difficulty often wastes valuable researcher and computational time by expending resources on uninteresting parts of data. Social sensors, or sensors which produce data based on human activity, such as Wikipedia, Twitter, and Facebook have an underlying structure which can be thought of as having a Power Law distribution. Such a distribution implies that few nodes generate large amounts of data. In this article, we propose a technique to take an arbitrary dataset and compute a power law distributed background model that bases its parameters on observed statistics. This model can be used to determine the suitability of using a power law or automatically identify high degree nodes for filtering and can be scaled to work with big data.