Object Proxy Patterns for Accelerating Distributed Applications

TL;DR

This paper introduces high-level proxy-based programming patterns—distributed futures, streaming, and ownership—that enhance distributed application performance by simplifying complex data flow management and optimizing resource usage.

Contribution

It proposes three novel high-level proxy patterns that extend the object proxy concept for more complex, dynamic distributed applications, supported by implementations and evaluations.

Findings

Substantial improvements in runtime, throughput, and memory usage in scientific applications.

Effective use of proxy patterns to optimize distributed data management.

Benchmarks demonstrate performance gains across multiple metrics.

Abstract

Workflow and serverless frameworks have empowered new approaches to distributed application design by abstracting compute resources. However, their typically limited or one-size-fits-all support for advanced data flow patterns leaves optimization to the application programmer -- optimization that becomes more difficult as data become larger. The transparent object proxy, which provides wide-area references that can resolve to data regardless of location, has been demonstrated as an effective low-level building block in such situations. Here we propose three high-level proxy-based programming patterns -- distributed futures, streaming, and ownership -- that make the power of the proxy pattern usable for more complex and dynamic distributed program structures. We motivate these patterns via careful review of application requirements and describe implementations of each pattern. We evaluate our implementations through a suite of benchmarks and by applying them in three substantial scientific applications, in which we demonstrate substantial improvements in runtime, throughput, and memory usage.