Em-K Indexing for Approximate Query Matching in Large-scale ER

TL;DR

This paper introduces Em-K Indexing, a novel approximate indexing method for entity resolution that uses spatial embeddings and Kd-trees to enable fast query matching on large-scale datasets.

Contribution

The paper proposes a new approximate indexing technique combining spatial embeddings and Kd-trees for efficient large-scale entity resolution query matching.

Findings

Effective in processing large datasets with reduced search space

Achieves query matching using only a small data fraction

Demonstrates promising results on multiple datasets

Abstract

Accurate and efficient entity resolution (ER) is a significant challenge in many data mining and analysis projects requiring integrating and processing massive data collections. It is becoming increasingly important in real-world applications to develop ER solutions that produce prompt responses for entity queries on large-scale databases. Some of these applications demand entity query matching against large-scale reference databases within a short time. We define this as the query matching problem in ER in this work. Indexing or blocking techniques reduce the search space and execution time in the ER process. However, approximate indexing techniques that scale to very large-scale datasets remain open to research. In this paper, we investigate the query matching problem in ER to propose an indexing method suitable for approximate and efficient query matching. We first use spatial mappings to embed records in a multidimensional Euclidean space that preserves the domain-specific similarity. Among the various mapping techniques, we choose multidimensional scaling. Then using a Kd-tree and the nearest neighbour search, the method returns a block of records that includes potential matches for a query. Our method can process queries against a large-scale dataset using only a fraction of the data $L$ (given the dataset size is $N$), with a $O(L^2)$ complexity where $L \ll N$. The experiments conducted on several datasets showed the effectiveness of the proposed method.