Fast Bayesian Record Linkage for Streaming Data Contexts

TL;DR

This paper introduces a fast Bayesian approach for streaming record linkage that efficiently updates link estimates as new data files arrive, outperforming traditional joint modeling methods in speed while maintaining accuracy.

Contribution

It generalizes a Bayesian Fellegi-Sunter model to multiple files and proposes two efficient streaming update methods for real-time record linkage.

Findings

Methods achieve near-equivalent accuracy to Gibbs sampling

Significant reduction in computational time

Effective in both simulated and real-world data

Abstract

Record linkage is the task of combining records from multiple files which refer to overlapping sets of entities when there is no unique identifying field. In streaming record linkage, files arrive sequentially in time and estimates of links are updated after the arrival of each file. This problem arises in settings such as longitudinal surveys, electronic health records, and online events databases, among others. The challenge in streaming record linkage is to efficiently update parameter estimates as new data arrive. We approach the problem from a Bayesian perspective with estimates calculated from posterior samples of parameters and present methods for updating link estimates after the arrival of a new file that are faster than fitting a joint model with each new data file. In this paper, we generalize a two-file Bayesian Fellegi-Sunter model to the multi-file case and propose two methods to perform streaming updates. We examine the effect of prior distribution on the resulting linkage accuracy as well as the computational trade-offs between the methods when compared to a Gibbs sampler through simulated and real-world survey panel data. We achieve near-equivalent posterior inference at a small fraction of the compute time. Supplemental materials for this article are available online.