Scaling Multiple-Source Entity Resolution using Statistically Efficient Transfer Learning

TL;DR

This paper introduces a transfer learning method for multi-source entity resolution that significantly reduces labeling costs and maintains high accuracy, addressing the quadratic data scaling challenge in heterogeneous data sources.

Contribution

The paper presents a novel transfer learning algorithm that efficiently shares structure across sources, reducing labeling requirements and computational costs in multi-source entity resolution.

Findings

Achieves constant precision/recall with linear labeling cost increase

Requires less training data than state-of-the-art methods

Maintains accuracy without runtime overhead

Abstract

We consider a serious, previously-unexplored challenge facing almost all approaches to scaling up entity resolution (ER) to multiple data sources: the prohibitive cost of labeling training data for supervised learning of similarity scores for each pair of sources. While there exists a rich literature describing almost all aspects of pairwise ER, this new challenge is arising now due to the unprecedented ability to acquire and store data from online sources, features driven by ER such as enriched search verticals, and the uniqueness of noisy and missing data characteristics for each source. We show on real-world and synthetic data that for state-of-the-art techniques, the reality of heterogeneous sources means that the number of labeled training data must scale quadratically in the number of sources, just to maintain constant precision/recall. We address this challenge with a brand new transfer learning algorithm which requires far less training data (or equivalently, achieves superior accuracy with the same data) and is trained using fast convex optimization. The intuition behind our approach is to adaptively share structure learned about one scoring problem with all other scoring problems sharing a data source in common. We demonstrate that our theoretically motivated approach incurs no runtime cost while it can maintain constant precision/recall with the cost of labeling increasing only linearly with the number of sources.