Kafka-ML: connecting the data stream with ML/AI frameworks

TL;DR

Kafka-ML is an open-source framework that integrates data streams with ML/AI pipelines using Apache Kafka, offering an accessible UI, containerization, and a novel data stream management approach to enhance real-time AI applications.

Contribution

It introduces Kafka-ML, a novel framework that connects data streams with ML/AI workflows, featuring a user-friendly interface, containerized deployment, and innovative data stream management.

Findings

Enables real-time ML/AI pipeline management via Kafka

Provides a containerized, portable, and fault-tolerant framework

Introduces a new approach for managing and reusing data streams

Abstract

Machine Learning (ML) and Artificial Intelligence (AI) have a dependency on data sources to train, improve and make predictions through their algorithms. With the digital revolution and current paradigms like the Internet of Things, this information is turning from static data into continuous data streams. However, most of the ML/AI frameworks used nowadays are not fully prepared for this revolution. In this paper, we proposed Kafka-ML, an open-source framework that enables the management of TensorFlow ML/AI pipelines through data streams (Apache Kafka). Kafka-ML provides an accessible and user-friendly Web User Interface where users can easily define ML models, to then train, evaluate and deploy them for inference. Kafka-ML itself and its deployed components are fully managed through containerization technologies, which ensure its portability and easy distribution and other features such as fault-tolerance and high availability. Finally, a novel approach has been introduced to manage and reuse data streams, which may lead to the (no) utilization of data storage and file systems.