Theoretical Limits of Record Linkage and Microclustering

TL;DR

This paper investigates the fundamental theoretical limits of record linkage and microclustering, demonstrating that accurate entity resolution is often impossible under realistic conditions, and emphasizes the need for more conservative and coarse inference methods.

Contribution

It provides the first theoretical analysis of the inherent difficulty of record linkage, especially in high-dimensional and low-separation scenarios, supported by simulations.

Findings

Entity resolution is fundamentally hard when entities are few relative to records.

Accurate population size estimation can be achieved even with poor entity resolution.

Results apply broadly beyond Gaussian models, highlighting the need for conservative inference.

Abstract

There has been substantial recent interest in record linkage, attempting to group the records pertaining to the same entities from a large database lacking unique identifiers. This can be viewed as a type of "microclustering," with few observations per cluster and a very large number of clusters. A variety of methods have been proposed, but there is a lack of literature providing theoretical guarantees on performance. We show that the problem is fundamentally hard from a theoretical perspective, and even in idealized cases, accurate entity resolution is effectively impossible when the number of entities is small relative to the number of records and/or the separation among records from different entities is not extremely large. To characterize the fundamental difficulty, we focus on entity resolution based on multivariate Gaussian mixture models, but our conclusions apply broadly and are supported by simulation studies inspired by human rights applications. These results suggest conservatism in interpretation of the results of record linkage, support collection of additional data to more accurately disambiguate the entities, and motivate a focus on coarser inference. For example, results from a simulation study suggest that sometimes one may obtain accurate results for population size estimation even when fine scale entity resolution is inaccurate.