The Design and Implementation of the Wave Transactional Filesystem

TL;DR

The Wave Transactional Filesystem (WTF) introduces transactional, POSIX-compatible file operations with a novel file slicing API, enabling efficient file transformations and high-performance distributed applications, outperforming traditional systems like HDFS.

Contribution

WTF is the first filesystem to combine transactional access with a new file slicing API, allowing efficient file transformations and improved performance in distributed environments.

Findings

WTF outperforms HDFS by up to four times in sorting benchmarks.

Microbenchmarks show modest overhead from new features.

WTF enables high-performance, consistent distributed file operations.

Abstract

This paper introduces the Wave Transactional Filesystem (WTF), a novel, transactional, POSIX-compatible filesystem based on a new file slicing API that enables efficient file transformations. WTF provides transactional access to a distributed filesystem, eliminating the possibility of inconsistencies across multiple files. Further, the file slicing API enables applications to construct files from the contents of other files without having to rewrite or relocate data. Combined, these enable a new class of high-performance applications. Experiments show that WTF can qualitatively outperform the industry-standard HDFS distributed filesystem, up to a factor of four in a sorting benchmark, by reducing I/O costs. Microbenchmarks indicate that the new features of WTF impose only a modest overhead on top of the POSIX-compatible API.