Design and Evaluation of a Simple Data Interface for Efficient Data Transfer Across Diverse Storage

TL;DR

This paper introduces Connector, a flexible data interface architecture that simplifies and optimizes data transfer across diverse storage systems, supporting third-party transfers and performance management.

Contribution

It presents a novel pluggable architecture for cross-storage data transfer, enabling secure, reliable, and efficient data exchange with performance modeling capabilities.

Findings

Supports third-party data transfers without involving third-party in data path.

Enables performance optimization and error handling during data transfers.

Provides a performance model for easy characterization without exhaustive benchmarking.

Abstract

Modern science and engineering computing environments often feature storage systems of different types, from parallel file systems in high-performance computing centers to object stores operated by cloud providers. To enable easy, reliable, secure, and performant data exchange among these different systems, we propose Connector, a pluggable data access architecture for diverse, distributed storage. By abstracting low-level storage system details, this abstraction permits a managed data transfer service (Globus in our case) to interact with a large and easily extended set of storage systems. Equally important, it supports third-party transfers: that is, direct data transfers from source to destination that are initiated by a third-party client but do not engage that third party in the data path. The abstraction also enables management of transfers for performance optimization, error handling, and end-to-end integrity. We present the Connector design, describe implementations for different storage services, evaluate tradeoffs inherent in managed vs.\ direct transfers, motivate recommended deployment options, and propose a performance model-based method that allows for easy characterization of performance in different contexts without exhaustive benchmarking.