Fuzzy Integration of Data Lake Tables

TL;DR

This paper introduces a fuzzy extension to the Full Disjunction operator for data integration, allowing approximate matches to improve dataset unification without significant performance loss.

Contribution

It proposes a novel data-driven method to incorporate fuzzy matching into Full Disjunction, addressing limitations of exact matching in diverse, real-world datasets.

Findings

Fuzzy Full Disjunction enhances data integration effectiveness.

The approach incurs minimal additional computational overhead.

Experimental results confirm improved dataset unification.

Abstract

Data integration is an important step in any data science pipeline where the objective is to unify the information available in different datasets for comprehensive analysis. Full Disjunction, which is an associative extension of the outer join operator, has been shown to be an effective operator for integrating datasets. It fully preserves and combines the available information. Existing Full Disjunction algorithms only consider the equi-join scenario where only tuples having the same value on joining columns are integrated. This, however, does not realistically represent an open data scenario, where datasets come from diverse sources with inconsistent values (e.g., synonyms, abbreviations, etc.) and with limited metadata. So, joining just on equal values severely limits the ability of Full Disjunction to fully combine datasets. Thus, in this work, we propose an extension of Full Disjunction to also account for "fuzzy" matches among tuples. We present a novel data-driven approach to enable the joining of approximate or fuzzy matches within Full Disjunction. Experimentally, we show that fuzzy Full Disjunction does not add significant time overhead over a state-of-the-art Full Disjunction implementation and also that it enhances the integration effectiveness.