Design and Evaluation of a Simple Data Interface for Efficient Data Transfer Across Diverse Storage
Zhengchun Liu, Rajkumar Kettimuthu, Joaquin Chung, Rachana, Ananthakrishnan, Michael Link, Ian Foster

TL;DR
This paper introduces Connector, a flexible data interface architecture that simplifies and optimizes data transfer across diverse storage systems, supporting third-party transfers and performance management.
Contribution
It presents a novel pluggable architecture for cross-storage data transfer, enabling secure, reliable, and efficient data exchange with performance modeling capabilities.
Findings
Supports third-party data transfers without involving third-party in data path.
Enables performance optimization and error handling during data transfers.
Provides a performance model for easy characterization without exhaustive benchmarking.
Abstract
Modern science and engineering computing environments often feature storage systems of different types, from parallel file systems in high-performance computing centers to object stores operated by cloud providers. To enable easy, reliable, secure, and performant data exchange among these different systems, we propose Connector, a pluggable data access architecture for diverse, distributed storage. By abstracting low-level storage system details, this abstraction permits a managed data transfer service (Globus in our case) to interact with a large and easily extended set of storage systems. Equally important, it supports third-party transfers: that is, direct data transfers from source to destination that are initiated by a third-party client but do not engage that third party in the data path. The abstraction also enables management of transfers for performance optimization, error…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
