SKALD: Scalable K-Anonymisation for Large Datasets

TL;DR

This paper introduces SKALD, a scalable algorithm for k-anonymisation that efficiently processes large datasets exceeding available RAM by leveraging chunk-based processing and sufficient statistics, improving performance and utility.

Contribution

SKALD is a novel algorithm that enables scalable k-anonymisation on large datasets by processing chunks with limited memory, outperforming standard methods.

Findings

Multi-fold performance improvement over existing methods

Effective processing of datasets exceeding RAM limits

Enhanced data utility post-anonymisation

Abstract

Data privacy and anonymisation are critical concerns in today's data-driven society, particularly when handling personal and sensitive user data. Regulatory frameworks worldwide recommend privacy-preserving protocols such as k-anonymisation to de-identify releases of tabular data. Available hardware resources provide an upper bound on the maximum size of dataset that can be processed at a time. Large datasets with sizes exceeding this upper bound must be broken up into smaller data chunks for processing. In these cases, standard k-anonymisation tools such as ARX can only operate on a per-chunk basis. This paper proposes SKALD, a novel algorithm for performing k-anonymisation on large datasets with limited RAM. Our SKALD algorithm offers multi-fold performance improvement over standard k-anonymisation methods by extracting and combining sufficient statistics from each chunk during processing to ensure successful k-anonymisation while providing better utility.