A Thorough Investigation of Content-Defined Chunking Algorithms for Data Deduplication

TL;DR

This paper provides a comprehensive theoretical and experimental comparison of leading Content-Defined Chunking algorithms for data deduplication, offering insights into their performance and limitations across multiple datasets.

Contribution

It offers the first thorough, impartial analysis and comparison of CDC algorithms, including new experimental results and contextual insights into their effectiveness.

Findings

Identifies strengths and weaknesses of various CDC algorithms.

Provides detailed performance metrics across multiple datasets.

Highlights previously unnoticed limitations of existing algorithms.

Abstract

Data deduplication emerged as a powerful solution for reducing storage and bandwidth costs in cloud settings by eliminating redundancies at the level of chunks. This has spurred the development of numerous Content-Defined Chunking (CDC) algorithms over the past two decades. Despite advancements, the current state-of-the-art remains obscure, as a thorough and impartial analysis and comparison is lacking. We conduct a rigorous theoretical analysis and impartial experimental comparison of several leading CDC algorithms. Using four realistic datasets, we evaluate these algorithms against four key metrics: throughput, deduplication ratio, average chunk size, and chunk-size variance. Our analyses, in many instances, extend the findings of their original publications by reporting new results and putting existing ones into context. Moreover, we highlight limitations that have previously gone unnoticed. Our findings provide valuable insights that inform the selection and optimization of CDC algorithms for practical applications in data deduplication.