Large-Scale Collective Entity Matching

TL;DR

This paper introduces a scalable framework for entity matching that enables applying machine learning algorithms to large datasets by dividing data into neighborhoods and exchanging information across them.

Contribution

It presents a novel, principled approach to scale generic entity matching algorithms using neighborhood-based message passing, with formal proofs and experimental validation.

Findings

Successfully scales EM algorithms to large datasets

Proves formal properties of the framework

Demonstrates effectiveness through experiments

Abstract

There have been several recent advancements in Machine Learning community on the Entity Matching (EM) problem. However, their lack of scalability has prevented them from being applied in practical settings on large real-life datasets. Towards this end, we propose a principled framework to scale any generic EM algorithm. Our technique consists of running multiple instances of the EM algorithm on small neighborhoods of the data and passing messages across neighborhoods to construct a global solution. We prove formal properties of our framework and experimentally demonstrate the effectiveness of our approach in scaling EM algorithms.