Theoretical Model and Practical Considerations for Data Lineage Reconstruction

TL;DR

This paper introduces a formal notation for modeling data flow and discusses its practical applications and potential programming paradigms to improve data lineage reconstruction.

Contribution

It presents a new formal data flow notation and evaluates its application in real-world data management scenarios.

Findings

Proposed a formal data flow modeling notation

Evaluated applications based on real-world use cases

Discussed potential programming paradigms for data lineage

Abstract

We live in a world driven by data. The amount of it outgrows anyone's ability to oversee it or even observe its scope. Along with all the advances in the space of data management, there is still a significant lack of formalism and standardization around defining data ecosystems and processes occurring within those. In order to address the issue we propose a notation for data flow modeling and evaluate some of the most common applications of it based on real-world use cases. To facilitate future work, we provide detailed reference of the data model we defined and consider potential programming paradigms.