Scalable Feature Matching Across Large Data Collections

TL;DR

This paper introduces fast, scalable algorithms for feature vector matching across large datasets by formulating the problem as a multidimensional assignment with decomposable costs, enabling efficient large-scale applications.

Contribution

The paper develops the first algorithms with linear time complexity and low storage for multidimensional feature matching, applicable to large datasets using squared Euclidean distance.

Findings

Algorithms outperform existing methods in speed and accuracy.

Linear scaling enables handling large datasets efficiently.

Successful application to a large neuroimaging database.

Abstract

This paper is concerned with matching feature vectors in a one-to-one fashion across large collections of datasets. Formulating this task as a multidimensional assignment problem with decomposable costs (MDADC), we develop extremely fast algorithms with time complexity linear in the number $n$ of datasets and space complexity a small fraction of the data size. These remarkable properties hinge on using the squared Euclidean distance as dissimilarity function, which can reduce ${n \choose 2}$ matching problems between pairs of datasets to $n$ problems and enable calculating assignment costs on the fly. To our knowledge, no other method applicable to the MDADC possesses these linear scaling and low-storage properties necessary to large-scale applications. In numerical experiments, the novel algorithms outperform competing methods and show excellent computational and optimization performances. An application of feature matching to a large neuroimaging database is presented. The algorithms of this paper are implemented in the R package matchFeat available at https://github.com/ddegras/matchFeat.