A Robust and Efficient Pipeline for Enterprise-Level Large-Scale Entity Resolution

TL;DR

This paper presents MERAI, a scalable and robust entity resolution pipeline capable of handling over 15 million records, outperforming existing tools in accuracy and efficiency for enterprise-level data management.

Contribution

Introduction of MERAI, a novel large-scale entity resolution pipeline that surpasses existing tools in scalability and accuracy for enterprise data management.

Findings

MERAI successfully processed datasets up to 15.7 million records.

MERAI achieved higher F1 scores than Dedupe and Splink.

MERAI demonstrated robustness and efficiency in large-scale experiments.

Abstract

Entity resolution (ER) remains a significant challenge in data management, especially when dealing with large datasets. This paper introduces MERAI (Massive Entity Resolution using AI), a robust and efficient pipeline designed to address record deduplication and linkage issues in high-volume datasets at an enterprise level. The pipeline's resilience and accuracy have been validated through various large-scale record deduplication and linkage projects. To evaluate MERAI's performance, we compared it with two well-known entity resolution libraries, Dedupe and Splink. While Dedupe failed to scale beyond 2 million records due to memory constraints, MERAI successfully processed datasets of up to 15.7 million records and produced accurate results across all experiments. Experimental data demonstrates that MERAI outperforms both baseline systems in terms of matching accuracy, with consistently higher F1 scores in both deduplication and record linkage tasks. MERAI offers a scalable and reliable solution for enterprise-level large-scale entity resolution, ensuring data integrity and consistency in real-world applications.