SMURF: Efficient and Scalable Metadata Access for Distributed Applications

TL;DR

SMURF is a scalable system that enhances distributed metadata access in cloud environments by combining novel pipelining, caching, and prefetching strategies, significantly reducing latency and improving performance based on real-world trace analysis.

Contribution

The paper introduces SMURF, a novel metadata access framework that leverages semantic locality and continuum caching to improve efficiency and scalability in distributed cloud applications.

Findings

Achieved 90% accuracy in prefetch prediction.

Reduced average fetch latency by 50%.

Improved cache hit rate through semantic locality-based prefetching.

Abstract

In parallel with big data processing and analysis dominating the usage of distributed and cloud infrastructures, the demand for distributed metadata access and transfer has increased. In many application domains, the volume of data generated exceeds petabytes, while the corresponding metadata amounts to terabytes or even more. This paper proposes a novel solution for efficient and scalable metadata access for distributed applications across wide-area networks, dubbed SMURF. Our solution combines novel pipelining and concurrent transfer mechanisms with reliability, provides distributed continuum caching and prefetching strategies to sidestep fetching latency, and achieves scalable and high-performance metadata fetch/prefetch services in the cloud. We also study the phenomenon of semantic locality in real trace logs, which is not well utilized in metadata access prediction. We implement a novel prefetch predictor based on this observation and compare it with three existing state-of-the-art prefetch schemes on Yahoo! Hadoop audit traces. By effectively caching and prefetching metadata based on the access patterns, our continuum caching and prefetching mechanism significantly improves local cache hit rate and reduces the average fetching latency. We replayed approximately 20 Million metadata access operations from real audit traces, in which our system achieved 90% accuracy during prefetch prediction and reduced the average fetch latency by 50% compared to the state-of-the-art mechanisms.