AutoStreamPipe: LLM Assisted Automatic Generation of Data Stream Processing Pipelines

TL;DR

AutoStreamPipe leverages Large Language Models and a Hypergraph of Thoughts to automate and improve the design, generation, and deployment of data stream processing pipelines, significantly reducing development time and errors.

Contribution

Introduces AutoStreamPipe, a framework that uses LLMs and HGoT to automate stream pipeline creation, bridging user intent and platform implementation.

Findings

Reduces development time by 6.3 times.

Reduces error rates by 5.19 times.

Demonstrates high accuracy in pipeline generation.

Abstract

Data pipelines are essential in stream processing as they enable the efficient collection, processing, and delivery of real-time data, supporting rapid data analysis. In this paper, we present AutoStreamPipe, a novel framework that employs Large Language Models (LLMs) to automate the design, generation, and deployment of stream processing pipelines. AutoStreamPipe bridges the semantic gap between high-level user intent and platform-specific implementations across distributed stream processing systems for structured multi-agent reasoning by integrating a Hypergraph of Thoughts (HGoT) as an extended version of GoT. AutoStreamPipe combines resilient execution strategies, advanced query analysis, and HGoT to deliver pipelines with good accuracy. Experimental evaluations on diverse pipelines demonstrate that AutoStreamPipe significantly reduces development time (x6.3) and error rates (x5.19), as measured by a novel Error-Free Score (EFS), compared to LLM code-generation methods.