# An Information-Theoretic Analysis of Deduplication

## TL;DR

This paper offers an information-theoretic framework for analyzing data deduplication, introducing new models and schemes, and demonstrating the near-optimality of a novel multi-chunk approach under mild assumptions.

## Contribution

It develops a formal information-theoretic analysis of deduplication, introduces a new multi-chunk scheme, and highlights the importance of boundary synchronization for optimality.

## Key findings

- Multi-chunk deduplication is order optimal under mild conditions.
- Boundary synchronization is crucial for deduplication efficiency.
- The paper formalizes fixed-length and variable-length deduplication schemes.

## Abstract

Deduplication finds and removes long-range data duplicates. It is commonly used in cloud and enterprise server settings and has been successfully applied to primary, backup, and archival storage. Despite its practical importance as a source-coding technique, its analysis from the point of view of information theory is missing. This paper provides such an information-theoretic analysis of data deduplication. It introduces a new source model adapted to the deduplication setting. It formalizes the two standard fixed-length and variable-length deduplication schemes, and it introduces a novel multi-chunk deduplication scheme. It then provides an analysis of these three deduplication variants, emphasizing the importance of boundary synchronization between source blocks and deduplication chunks. In particular, under fairly mild assumptions, the proposed multi-chunk deduplication scheme is shown to be order optimal.