Efficient Row-Level Lineage Leveraging Predicate Pushdown

TL;DR

PredTrace is a novel lineage inference method that leverages predicate pushdown to efficiently and accurately determine row-level lineage across diverse data pipelines, outperforming previous lazy approaches in speed and coverage.

Contribution

It introduces PredTrace, a new approach that combines the advantages of eager and lazy lineage tracking using predicate pushdown for better coverage, efficiency, and adaptability.

Findings

Achieves higher coverage on TPC-H and real-world pipelines.

Infers lineage in seconds, up to 10x faster than prior methods.

Provides precise or approximate lineage depending on intermediate result availability.

Abstract

Row-level lineage explains what input rows produce an output row through a data processing pipeline, having many applications like data debugging, auditing, data integration, etc. Prior work on lineage falls in two lines: eager lineage tracking and lazy lineage inference. Eager tracking integrates lineage tracing tightly into the operator implementation, enabling efficient customized tracking. However, this approach is intrusive, system-specific, and lacks adaptability. In contrast, lazy inference generates additional queries to compute lineage; it can be easily applied to any database, but the lineage query is usually slow. Furthermore, both approaches have limited coverage of the type of data processing pipeline supported due to operator-specific tracking or inference rules. In this work, we propose PredTrace, a lineage inference approach that achieves easy adaptation, low runtime overhead, efficient lineage querying, and high pipeline coverage. It achieves this by leveraging predicate pushdown: pushing a row-selection predicate that describes the target output down to source tables and querying the lineage by running the pushed-down predicate. PredTrace may require saving intermediate results when running the pipeline in order to compute the precise lineage. When this is not viable, it can still infer lineage but may return a superset. Compared to prior work, PredTrace achieves higher coverage on TPC-H queries as well as 70 sampled real-world data processing pipelines in which UDFs are widely used. It can infer lineage in seconds, outperforming prior lazy approaches by up to 10x.