Effectively Testing System Configurations of Critical IoT Analytics Pipelines

TL;DR

This paper presents Timon, a Kubernetes-integrated tool for efficiently testing and optimizing configurations of IoT analytics pipelines, demonstrated on a traffic monitoring application to improve decision-making in critical infrastructures.

Contribution

The paper introduces Timon, a novel approach and prototype for systematic testing and optimization of IoT analytics pipeline configurations in distributed systems.

Findings

Timon enables parallel pipeline replication and evaluation.

Optimized configurations improve system performance for critical IoT applications.

The approach is validated on a traffic monitoring use case.

Abstract

The emergence of the Internet of Things has seen the introduction of numerous connected devices used for the monitoring and control of even Critical Infrastructures. Distributed stream processing has become key to analyzing data generated by these connected devices and improving our ability to make decisions. However, optimizing these systems towards specific Quality of Service targets is a difficult and time-consuming task, due to the large-scale distributed systems involved, the existence of so many configuration parameters, and the inability to easily determine the impact of tuning these parameters. In this paper we present an approach for the effective testing of system configurations for critical IoT analytics pipelines. We demonstrate our approach with a prototype that we called Timon which is integrated with Kubernetes. This tool allows pipelines to be easily replicated in parallel and evaluated to determine the optimal configuration for specific applications. We demonstrate the usefulness of our approach by investigating different configurations of an exemplary geographically-based traffic monitoring application implemented in Apache Flink.