A Comparison of Blocking Methods for Record Linkage

TL;DR

This paper compares traditional blocking and locality sensitive hashing methods for record linkage, analyzing their effectiveness, efficiency, and privacy considerations across various synthetic datasets.

Contribution

It provides a comprehensive comparison of blocking techniques and locality sensitive hashing variants, highlighting their strengths and limitations for record linkage.

Findings

Traditional blocking offers high recall but lower reduction ratios.

Locality sensitive hashing achieves better reduction ratios with comparable recall.

Privacy considerations vary between methods, impacting their suitability for sensitive data.

Abstract

Record linkage seeks to merge databases and to remove duplicates when unique identifiers are not available. Most approaches use blocking techniques to reduce the computational complexity associated with record linkage. We review traditional blocking techniques, which typically partition the records according to a set of field attributes, and consider two variants of a method known as locality sensitive hashing, sometimes referred to as "private blocking." We compare these approaches in terms of their recall, reduction ratio, and computational complexity. We evaluate these methods using different synthetic datafiles and conclude with a discussion of privacy-related issues.