# A Robust Fault-Tolerant and Scalable Cluster-wide Deduplication for Shared-Nothing Storage Systems

## TL;DR

This paper introduces a fault-tolerant, scalable cluster-wide deduplication method for shared-nothing storage systems, enhancing space efficiency while ensuring consistency and robustness against failures.

## Contribution

It presents a novel distributed deduplication metadata shard design and an asynchronous consistency mechanism tailored for shared-nothing storage architectures.

## Key findings

- Achieves high disk-space savings with minimal performance impact.
- Demonstrates robustness during sudden server failures.
- Maintains scalability and consistency in distributed environments.

## Abstract

Deduplication has been largely employed in distributed storage systems to improve space efficiency. Traditional deduplication research ignores the design specifications of shared-nothing distributed storage systems such as no central metadata bottleneck, scalability, and storage rebalancing. Further, deduplication introduces transactional changes, which are prone to errors in the event of a system failure, resulting in inconsistencies in data and deduplication metadata. In this paper, we propose a robust, fault-tolerant and scalable cluster-wide deduplication that can eliminate duplicate copies across the cluster. We design a distributed deduplication metadata shard which guarantees performance scalability while preserving the design constraints of shared- nothing storage systems. The placement of chunks and deduplication metadata is made cluster-wide based on the content fingerprint of chunks. To ensure transactional consistency and garbage identification, we employ a flag-based asynchronous consistency mechanism. We implement the proposed deduplication on Ceph. The evaluation shows high disk-space savings with minimal performance degradation as well as high robustness in the event of sudden server failure.