Building User-defined Runtime Adaptation Routines for Stream Processing Applications

TL;DR

This paper introduces an orchestrator component for stream processing applications that enables automatic runtime adaptation through user-defined routines, improving flexibility and responsiveness to system and data changes.

Contribution

The paper presents a novel orchestrator framework allowing developers to create custom runtime adaptation routines for stream processing applications.

Findings

Successfully integrated orchestrator in IBM System S

Demonstrated adaptation to data distribution changes

Enabled dynamic application composition

Abstract

Stream processing applications are deployed as continuous queries that run from the time of their submission until their cancellation. This deployment mode limits developers who need their applications to perform runtime adaptation, such as algorithmic adjustments, incremental job deployment, and application-specific failure recovery. Currently, developers do runtime adaptation by using external scripts and/or by inserting operators into the stream processing graph that are unrelated to the data processing logic. In this paper, we describe a component called orchestrator that allows users to write routines for automatically adapting the application to runtime conditions. Developers build an orchestrator by registering and handling events as well as specifying actuations. Events can be generated due to changes in the system state (e.g., application component failures), built-in system metrics (e.g., throughput of a connection), or custom application metrics (e.g., quality score). Once the orchestrator receives an event, users can take adaptation actions by using the orchestrator actuation APIs. We demonstrate the use of the orchestrator in IBM's System S in the context of three different applications, illustrating application adaptation to changes on the incoming data distribution, to application failures, and on-demand dynamic composition.