SNIP: An Adaptation of Sorted Neighborhood Methods for Deduplicating Pedigree Data

TL;DR

This paper introduces SNIP, an unsupervised algorithm based on sorted neighborhood methods, to efficiently identify and remove duplicate pedigree records in large clinical datasets, improving data quality for hereditary disease research.

Contribution

The paper presents SNIP, a novel adaptation of sorted neighborhood algorithms specifically designed for complex pedigree data deduplication, with demonstrated effectiveness on real-world datasets.

Findings

SNIP accurately detects pedigree duplicates in simulated data.

Application to real data uncovered large duplicate clusters, removing 33% of pedigrees.

The method improves data quality for hereditary disease analysis.

Abstract

Pedigree data contain family history information that is used to analyze hereditary diseases. These clinical data sets may contain duplicate records due to the same family visiting a clinic multiple times or a clinician entering multiple versions of the family for testing purposes. Inferences drawn from the data or using them for training or validation without removing the duplicates could lead to invalid conclusions, and hence identifying the duplicates is essential. Since family structures can be complex, existing deduplication algorithms cannot be applied directly. We first motivate the importance of deduplication by examining the impact of pedigree duplicates on the training and validation of a familial risk prediction model. We then introduce an unsupervised algorithm, which we call SNIP (Sorted NeIghborhood for Pedigrees), that builds on the sorted neighborhood method to efficiently find and classify pairwise comparisons by leveraging the inherent hierarchical nature of the pedigrees. We conduct a simulation study to assess the performance of the algorithm and find parameter configurations where the algorithm is able to accurately detect the duplicates. We then apply the method to data from the Risk Service, which includes over 300,000 pedigrees at high risk of hereditary cancers, and uncover large clusters of potential duplicate families. After removing 104,520 pedigrees (33% of original data), the resulting Risk Service dataset can now be used for future analysis, training, and validation. The algorithm is available as an R package snipR available at https://github.com/bayesmendel/snipR.