A Novel Micro-service Based Platform for Composition, Deployment and Execution of BDA Applications

TL;DR

This paper introduces ALIDA, a micro-service platform designed for composing, deploying, and executing Big Data Analytics workflows, enabling easier interaction for developers and data analysts across various domains.

Contribution

It presents a unified, extensible platform that supports application registration, workflow creation, and auto-tuning of Big Data frameworks, with practical integration in the avionic industry.

Findings

Successful deployment in avionic data analysis

Enhanced workflow creation for data analysts

Auto-tuning improves analytics performance

Abstract

Big Data are growing at an exponential rate and it becomes necessary the use of tools and technologies to manage, process and visualize them in order to extract value. In this paper a micro-service based platform is presented for the composition, deployment and execution of Big Data Analytics (BDA) application workflows in several domains and scenarios is presented. ALIDA is a result coming from previous research activities by ENGINEERING. It aims to achieve a unified platform that allows both BDA application developers and data analysts to interact with it. Developers will be able to register new BDA applications through the exposed API and/or through the web user interface. Data analysts will be able to use the BDA applications provided to create batch/stream workflows through a dashboard user interface to manipulate and subsequently visualize results from one or more sources. The platform also supports the auto-tuning of Big Data frameworks deployment properties to improve metrics for analytics application. ALIDA has been properly extended and integrated into a software solution for the analysis of large amounts of data from the avionic industries. A use case within this context is then presented.