Large scale deduplication based on fingerprints

TL;DR

This paper introduces a novel fingerprint deduplication algorithm that significantly reduces computational complexity from quadratic to linear time, enabling large-scale databases to be processed efficiently on a single computer.

Contribution

The paper presents a new fingerprint indexing method using a 5x5 matrix that allows for fast clustering and deduplication, improving scalability and speed over existing algorithms.

Findings

Achieves less than 1% penetration rate in deduplication.

Performs deduplication of 10 million fingerprints in under two hours.

Reduces computational complexity from O(n^2) to O(n) for large datasets.

Abstract

In fingerprint-based systems, the size of databases increases considerably with population growth. In developing countries, because of the difficulty in using a central system when enlisting voters, it often happens that several regional voter databases are created and then merged to form a central database. A process is used to remove duplicates and ensure uniqueness by voter. Until now, companies specializing in biometrics use several costly computing servers with algorithms to perform large-scale deduplication based on fingerprints. These algorithms take a considerable time because of their complexity in O (n2), where n is the size of the database. This article presents an algorithm that can perform this operation in O (2n), with just a computer. It is based on the development of an index obtained using a 5 * 5 matrix performed on each fingerprint. This index makes it possible to build clusters of O (1) in size in order to compare fingerprints. This approach has been evaluated using close to 11 4000 fingerprints, and the results obtained show that this approach allows a penetration rate of less than 1%, an almost O (1) identification, and an O (n) deduplication. A base of 10 000 000 fingerprints can be deduplicated with a just computer in less than two hours, contrary to several days and servers for the usual tools. Keywords: fingerprint, cluster, index, deduplication.