Redundancy-Driven Top-$k$ Functional Dependency Discovery

TL;DR

This paper introduces SDP, a novel method for efficiently discovering the top-k functional dependencies in large datasets by leveraging redundancy-based pruning to reduce computational complexity and result size.

Contribution

The paper presents SDP, a new algorithm that efficiently finds the most redundant functional dependencies using pruning techniques based on monotone upper bounds.

Findings

SDP significantly outperforms exhaustive methods in speed and memory usage.

SDP effectively identifies the top-k FDs ranked by redundancy.

Experiments on 40 datasets demonstrate SDP's scalability and efficiency.

Abstract

Functional dependencies (FDs) are basic constraints in relational databases and are used for many data management tasks. Most FD discovery algorithms find all valid dependencies, but this causes two problems. First, the computational cost is prohibitive: computational complexity grows quadratically with the number of tuples and exponentially with the number of attributes, making discovery slow on large-scale and high-dimensional data. Second, the result set can be huge, making it hard to identify useful dependencies. We propose SDP (Selective-Discovery-and-Prune), which discovers the top-$k$ FDs ranked by redundancy count. Redundancy count measures how much duplicated information an FD explains and connects directly to storage overhead and update anomalies. SDP uses an upper bound on redundancy to prune the search space. It is proved that this upper bound is monotone: adding attributes refines partitions and thus decreases the bound. Once the bound falls below the top-$k$ threshold, the entire branch can be skipped. We improve SDP with three optimizations: ordering attributes by partition cardinality, using pairwise statistics in a Partition Cardinality Matrix to tighten bounds, and a global scheduler to explore promising branches first. Experiments on over 40 datasets show that SDP is much faster and uses less memory than exhaustive methods.