Provenance-aware Discovery of Functional Dependencies on Integrated Views

TL;DR

This paper introduces InFine, a novel method for efficiently discovering functional dependencies in integrated database views by leveraging provenance information, logical inference, and selective mining, significantly reducing computation time and memory usage.

Contribution

It presents the first approach to infer FDs for integrated views using only base relation dependencies, avoiding full view computation, and introduces algorithms for faster, on-the-fly FD discovery.

Findings

InFine outperforms traditional methods by 10-100x in runtime.

InFine is more memory-efficient while preserving FD provenance.

The approach effectively infers most FDs without full view recomputation.

Abstract

The automatic discovery of functional dependencies(FDs) has been widely studied as one of the hardest problems in data profiling. Existing approaches have focused on making the FD computation efficient while inspecting single relations at a time. In this paper, for the first time we address the problem of inferring FDs for multiple relations as they occur in integrated views by solely using the functional dependencies of the base relations of the view itself. To this purpose, we leverage logical inference and selective mining and show that we can discover most of the exact FDs from the base relations and avoid the full computation of the FDs for the integrated view itself, while at the same time preserving the lineage of FDs of base relations. We propose algorithms to speedup the inferred FD discovery process and mine FDs on-the-fly only from necessary data partitions. We present InFine(INferred FunctIoNal dEpendency), an end-to-end solution to discover inferred FDs on integrated views by leveraging provenance information of base relations. Our experiments on a range of real-world and synthetic datasets demonstrate the benefits of our method over existing FD discovery methods that need to rerun the discovery process on the view from scratch and cannot exploit lineage information on the FDs. We show that InFine outperforms traditional methods necessitating the full integrated view computation by one to two order of magnitude in terms of runtime. It is also the most memory efficient method while preserving FD provenance information using mainly inference from base table with negligible execution time.