Accelerating Communications in Federated Applications with Transparent Object Proxies

TL;DR

This paper introduces ProxyStore, a system that uses transparent object proxies to decouple control and data flow in federated applications, enabling efficient, unilateral data communication across distributed systems.

Contribution

It presents a novel programming paradigm extending pass-by-reference with object proxies for transparent, efficient data sharing in distributed environments.

Findings

Improved data transfer efficiency in federated systems.

Reduced costs in cloud and edge computing workflows.

Enhanced data communication transparency and unilateral updates.

Abstract

Advances in networks, accelerators, and cloud services encourage programmers to reconsider where to compute -- such as when fast networks make it cost-effective to compute on remote accelerators despite added latency. Workflow and cloud-hosted serverless computing frameworks can manage multi-step computations spanning federated collections of cloud, high-performance computing (HPC), and edge systems, but passing data among computational steps via cloud storage can incur high costs. Here, we overcome this obstacle with a new programming paradigm that decouples control flow from data flow by extending the pass-by-reference model to distributed applications. We describe ProxyStore, a system that implements this paradigm by providing object proxies that act as wide-area object references with just-in-time resolution. This proxy model enables data producers to communicate data unilaterally, transparently, and efficiently to both local and remote consumers. We demonstrate the benefits of this model with synthetic benchmarks and real-world scientific applications, running across various computing platforms.