Multi-tenant Pub/Sub Processing for Real-time Data Streams

TL;DR

This paper presents a dynamic runtime system for multi-tenant stream processing that allows multiple users to deploy custom data-processing services using a subscription-based model, enabling flexible real-time data analytics.

Contribution

It introduces a novel runtime for constructing dynamic, user-defined data stream processing topologies in a multi-tenant environment, supporting flexible real-time analytics.

Findings

Supports multiple users deploying custom services

Enables dynamic construction of processing topologies

Facilitates real-time data stream processing

Abstract

Devices and sensors generate streams of data across a diversity of locations and protocols. That data usually reaches a central platform that is used to store and process the streams. Processing can be done in real time, with transformations and enrichment happening on-the-fly, but it can also happen after data is stored and organized in repositories. In the former case, stream processing technologies are required to operate on the data; in the latter batch analytics and queries are of common use. This paper introduces a runtime to dynamically construct data stream processing topologies based on user-supplied code. These dynamic topologies are built on-the-fly using a data subscription model defined by the applications that consume data. Each user-defined processing unit is called a Service Object. Every Service Object consumes input data streams and may produce output streams that others can consume. The subscription-based programing model enables multiple users to deploy their own data-processing services. The runtime does the dynamic forwarding of data and execution of Service Objects from different users. Data streams can originate in real-world devices or they can be the outputs of Service Objects.