d-blink: Distributed End-to-End Bayesian Entity Resolution

TL;DR

d-blink introduces a scalable, distributed Bayesian model for entity resolution that jointly performs blocking and record linkage, maintaining posterior accuracy while handling large noisy databases efficiently.

Contribution

It proposes a novel distributed Bayesian ER framework that combines blocking and linkage without sacrificing posterior correctness, using auxiliary variables and efficient Gibbs sampling.

Findings

Successfully scales to large datasets including census data

Maintains accurate posterior inference despite data partitioning

Demonstrates effectiveness across six diverse datasets

Abstract

Entity resolution (ER; also known as record linkage or de-duplication) is the process of merging noisy databases, often in the absence of unique identifiers. A major advancement in ER methodology has been the application of Bayesian generative models, which provide a natural framework for inferring latent entities with rigorous quantification of uncertainty. Despite these advantages, existing models are severely limited in practice, as standard inference algorithms scale quadratically in the number of records. While scaling can be managed by fitting the model on separate blocks of the data, such a na\"ive approach may induce significant error in the posterior. In this paper, we propose a principled model for scalable Bayesian ER, called "distributed Bayesian linkage" or d-blink, which jointly performs blocking and ER without compromising posterior correctness. Our approach relies on several key ideas, including: (i) an auxiliary variable representation that induces a partition of the entities and records into blocks; (ii) a method for constructing well-balanced blocks based on k-d trees; (iii) a distributed partially-collapsed Gibbs sampler with improved mixing; and (iv) fast algorithms for performing Gibbs updates. Empirical studies on six data sets---including a case study on the 2010 Decennial Census---demonstrate the scalability and effectiveness of our approach.