Distributed Recoverable Sketches (Extended Version)

TL;DR

This paper introduces a modular framework for distributed, recoverable sketches in network environments, focusing on frequency estimation sketches like Count-Min Sketch, balancing space, runtime, and recovery traffic.

Contribution

It presents a novel, scalable system for cooperative sketch recovery after node crashes, with flexible update strategies and a pluggable architecture for various data structures.

Findings

Effective recovery of sketches after node failures

Comparison of full vs incremental updates for efficiency

Design guidelines balancing space, runtime, and traffic

Abstract

Sketches are commonly used in computer systems and network monitoring tools to provide efficient query executions while maintaining a compact data representation. Switches and routers maintain sketches to track statistical characteristics of network traffic. The availability of such data is essential for the network analysis as a whole. Consequently, being able to recover sketches is critical after a switch crash. In this work, we explore how nodes in a network environment can cooperate to recover sketch data whenever any subset of them crashes. In particular, we focus on frequency estimation linear sketches, such as the Count-Min Sketch. We consider various approaches to ensure data reliability and explore the trade-offs between space consumption, runtime overheads, and traffic during recovery, which we point out as design guidelines. Besides different aspects of efficacy, we design a modular system for ease of maintenance and further scaling. A key aspect we examine is how the nodes update each other regarding their sketch content as it evolves over time. In particular, we compare periodic full updates vs incremental updates. We also examine several data structures to economically represent and encode a batch of latest changes. Our framework is generic, and other data structures can be plugged-in via an abstract API as long as they implement the corresponding API methods.