Distributed Management of Massive Data: an Efficient Fine-Grain Data Access Scheme

TL;DR

This paper presents a novel distributed data management scheme that enables efficient fine-grain access to massive data blocks in large-scale environments, using RAM-based storage and a DHT-based metadata system for transparent data access.

Contribution

It introduces a distributed, RAM-based storage system with a DHT-based metadata scheme that allows transparent, fine-grain data access in large-scale distributed settings.

Findings

Prototype implementation demonstrates promising performance.

Efficient fine-grain data access achieved in large-scale environments.

Supports transparent data access via global identifiers.

Abstract

This paper addresses the problem of efficiently storing and accessing massive data blocks in a large-scale distributed environment, while providing efficient fine-grain access to data subsets. This issue is crucial in the context of applications in the field of databases, data mining and multimedia. We propose a data sharing service based on distributed, RAM-based storage of data, while leveraging a DHT-based, natively parallel metadata management scheme. As opposed to the most commonly used grid storage infrastructures that provide mechanisms for explicit data localization and transfer, we provide a transparent access model, where data are accessed through global identifiers. Our proposal has been validated through a prototype implementation whose preliminary evaluation provides promising results.