Mining CFD Rules on Big Data

TL;DR

This paper presents a scalable approach for discovering conditional functional dependencies (CFDs) on big data by using sampling, fault-tolerance, and conflict resolution techniques, enabling effective rule discovery on billion-tuple datasets.

Contribution

It introduces a comprehensive framework combining sampling, fault-tolerance, and parameter tuning for CFD discovery tailored to large, low-quality datasets.

Findings

Effective CFD rules discovered on billion-tuple data

Method reduces computational time significantly

Framework handles low-quality and voluminous data

Abstract

Current conditional functional dependencies (CFDs) discovery algorithms always need a well-prepared training data set. This makes them difficult to be applied on large datasets which are always in low-quality. To handle the volume issue of big data, we develop the sampling algorithms to obtain a small representative training set. For the low-quality issue of big data, we then design the fault-tolerant rule discovery algorithm and the conflict resolution algorithm. We also propose parameter selection strategy for CFD discovery algorithm to ensure its effectiveness. Experimental results demonstrate that our method could discover effective CFD rules on billion-tuple data within reasonable time.