Big Data Refinement

TL;DR

This paper investigates how the concept of data refinement from formal methods can be applied to understand and improve big data processing techniques.

Contribution

It explores the applicability of data refinement principles to big data, bridging formal methods and data processing paradigms for better understanding.

Findings

Data refinement offers a useful perspective on big data processing

Formal methods can inform big data refinement strategies

Potential for improved data processing frameworks

Abstract

"Big data" has become a major area of research and associated funding, as well as a focus of utopian thinking. In the still growing research community, one of the favourite optimistic analogies for data processing is that of the oil refinery, extracting the essence out of the raw data. Pessimists look for their imagery to the other end of the petrol cycle, and talk about the "data exhausts" of our society. Obviously, the refinement community knows how to do "refining". This paper explores the extent to which notions of refinement and data in the formal methods community relate to the core concepts in "big data". In particular, can the data refinement paradigm can be used to explain aspects of big data processing?