Using Dedicated and Opportunistic Networks in Synergy for a Cost-effective Distributed Stream Processing Platform

TL;DR

This paper proposes a hybrid distributed stream processing platform leveraging both dedicated and opportunistic networks, demonstrating through simulations that this synergy improves throughput and resource utilization cost-effectively.

Contribution

It introduces a system model and decentralized algorithms for managing hybrid network resources in distributed stream processing, enhancing throughput and efficiency.

Findings

Hybrid resource management increases task throughput.

Synergy reduces reliance on costly dedicated links.

Simulation results show improved resource utilization.

Abstract

This paper presents a case for exploiting the synergy of dedicated and opportunistic network resources in a distributed hosting platform for data stream processing applications. Our previous studies have demonstrated the benefits of combining dedicated reliable resources with opportunistic resources in case of high-throughput computing applications, where timely allocation of the processing units is the primary concern. Since distributed stream processing applications demand large volume of data transmission between the processing sites at a consistent rate, adequate control over the network resources is important here to assure a steady flow of processing. In this paper, we propose a system model for the hybrid hosting platform where stream processing servers installed at distributed sites are interconnected with a combination of dedicated links and public Internet. Decentralized algorithms have been developed for allocation of the two classes of network resources among the competing tasks with an objective towards higher task throughput and better utilization of expensive dedicated resources. Results from extensive simulation study show that with proper management, systems exploiting the synergy of dedicated and opportunistic resources yield considerably higher task throughput and thus, higher return on investment over the systems solely using expensive dedicated resources.