EAIFD: A Fast and Scalable Algorithm for Incremental Functional Dependency Discovery

TL;DR

EAIFD is a novel incremental functional dependency discovery algorithm that significantly improves speed and memory efficiency by using hypergraph-based enumeration and innovative data structures, outperforming existing methods.

Contribution

The paper introduces EAIFD, a scalable incremental FD discovery algorithm with novel hypergraph and hash table techniques that reduce runtime and memory usage.

Findings

Up to 10x faster than existing algorithms.

Reduces memory consumption by over 100x.

Effective on real-world datasets.

Abstract

Functional dependencies (FDs) are fundamental integrity constraints in relational databases, but discovering them under incremental updates remains challenging. While static algorithms are inefficient due to full re-execution, incremental algorithms suffer from severe performance and memory bottlenecks. To address these challenges, this paper proposes EAIFD, a novel algorithm for incremental FD discovery. EAIFD maintains the partial hypergraph of difference sets and reframes the incremental FD discovery problem into minimal hitting set enumeration on hypergraph, avoiding full re-runs. EAIFD introduces two key innovations. First, a multi-attribute hash table ($MHT$) is devised for high-frequency key-value mappings of valid FDs, whose memory consumption is proven to be independent of the dataset size. Second, two-step validation strategy is developed to efficiently validate the enumerated candidates, which leverages $MHT$ to effectively reduce the validation space and then selectively loads data blocks for batch validation of remaining candidates, effectively avoiding repeated I/O operations. Experimental results on real-world datasets demonstrate the significant advantages of EAIFD. Compared to existing algorithms, EAIFD achieves up to an order-of-magnitude speedup in runtime while reducing memory usage by over two orders-of-magnitude, establishing it as a highly efficient and scalable solution for incremental FD discovery.