Efficiently Transforming Tables for Joinability

TL;DR

This paper presents an efficient algorithm for transforming and joining textual data from different sources despite formatting differences, significantly improving speed over existing methods.

Contribution

We develop a novel algorithm that efficiently finds transformations to join mismatched textual data, outperforming state-of-the-art approaches in coverage and speed.

Findings

Algorithm covers all transformations of the baseline approach.

Our method is several orders of magnitude faster.

Effective on both real and synthetic datasets.

Abstract

Data from different sources rarely conform to a single formatting even if they describe the same set of entities, and this raises concerns when data from multiple sources must be joined or cross-referenced. Such a formatting mismatch is unavoidable when data is gathered from various public and third-party sources. Commercial database systems are not able to perform the join when there exist differences in data representation or formatting, and manual reformatting is both time consuming and error-prone. We study the problem of efficiently joining textual data under the condition that the join columns are not formatted the same and cannot be equi-joined, but they become joinable under some transformations. The problem is challenging simply because the number of possible transformations explodes with both the length of the input and the number of rows, even if each transformation is formed using very few basic units. We show that an efficient algorithm can be developed based on the common characteristics of the joined columns, and develop one such algorithm over a rich set of basic operations that can be composed to form transformations. We compare both the coverage and the running time of our algorithm to a state-of-the-art approach, and show that our algorithm covers every transformation that is covered in the state-of-the-art approach but is a few orders of magnitude faster, as evaluated on various real and synthetic data.