Synchronizing Many Filesystems in Near Linear Time

TL;DR

This paper presents a provably correct, near-linear time synchronization algorithm for multiple filesystem replicas, capable of handling complex structures including DAGs, with asynchronous operation support.

Contribution

It introduces a novel algebraic approach to filesystem synchronization that achieves subquadratic complexity and supports arbitrary synchronized states and complex filesystem structures.

Findings

Synchronization operates in linear space and time after sorting.

Algorithm correctly identifies and resolves conflicting commands.

Supports asynchronous synchronization and complex filesystem structures.

Abstract

Finding a provably correct subquadratic synchronization algorithm for many filesystem replicas is one of the main theoretical problems in Operational Transformation (OT) and Conflict-free Replicated Data Types (CRDT) frameworks. Based on the Algebraic Theory of Filesystems, which incorporates non-commutative filesystem commands natively, we developed and built a proof-of-concept implementation of an algorithm suite which synchronizes an arbitrary number of replicas. The result is provably correct, and the synchronized system is created in linear space and time after an initial sorting phase. It works by identifying conflicting command pairs and requesting one of the commands to be removed. The method can be guided to reach any of the theoretically possible synchronized states. The algorithm also allows asynchronous usage. After the client sends a synchronization request, the local replica remains available for further modifications. When the synchronization instructions arrive, they can be merged with the changes made since the synchronization request. The suite also works on filesystems with directed acyclic graph-based path structure in place of the traditional tree-like arrangement. Consequently, our algorithms apply to filesystems with hard or soft links as long as the links create no loops.