DynoStore: A wide-area distribution system for the management of data over heterogeneous storage

TL;DR

DynoStore is a system that manages data across heterogeneous storage systems using standardized interfaces, load balancing, and erasure coding, improving performance and fault tolerance in wide-area distributed environments.

Contribution

It introduces data containers as a unified abstraction for heterogeneous storage management and demonstrates improved performance and resilience over existing solutions.

Findings

10% performance improvement over centralized cloud systems

Superior fault tolerance compared to traditional systems

Effective management of medical and satellite data across distributed environments

Abstract

Data distribution across different facilities offers benefits such as enhanced resource utilization, increased resilience through replication, and improved performance by processing data near its source. However, managing such data is challenging due to heterogeneous access protocols, disparate authentication models, and the lack of a unified coordination framework. This paper presents DynoStore, a system that manages data across heterogeneous storage systems. At the core of DynoStore are data containers, an abstraction that provides standardized interfaces for seamless data management, irrespective of the underlying storage systems. Multiple data container connections create a cohesive wide-area storage network, ensuring resilience using erasure coding policies. Furthermore, a load-balancing algorithm ensures equitable and efficient utilization of storage resources. We evaluate DynoStore using benchmarks and real-world case studies, including the management of medical and satellite data across geographically distributed environments. Our results demonstrate a 10\% performance improvement compared to centralized cloud-hosted systems while maintaining competitive performance with state-of-the-art solutions such as Redis and IPFS. DynoStore also exhibits superior fault tolerance, withstanding more failures than traditional systems.