Building Analytics Pipelines for Querying Big Streams and Data Histories with H-STREAM

TL;DR

H-STREAM is a cloud-based engine that enables flexible, scalable analysis and visualization of IoT data streams through micro-services, effectively handling high velocity and volume in real-world scenarios.

Contribution

The paper introduces H-STREAM, a novel micro-service based framework for processing and visualizing big IoT data streams with adaptive techniques and cloud deployment.

Findings

Successfully processed high-volume IoT streams in neuroscience and social connectivity scenarios.

Demonstrated scalability and adaptability of H-STREAM in cloud environments.

Validated effectiveness in handling velocity and volume challenges.

Abstract

This paper introduces H-STREAM, a big stream/data processing pipelines evaluation engine that proposes stream processing operators as micro-services to support the analysis and visualisation of Big Data streams stemming from IoT (Internet of Things) environments. H-STREAM micro-services combine stream processing and data storage techniques tuned depending on the number of things producing streams, the pace at which they produce them, and the physical computing resources available for processing them online and delivering them to consumers. H-STREAM delivers stream processing and visualisation micro-services installed in a cloud environment. Micro-services can be composed for implementing specific stream aggregation analysis pipelines as queries. The paper presents an experimental validation using Microsoft Azure as a deployment environment for testing the capacity of H-STREAM for dealing with velocity and volume challenges in an (i) a neuroscience experiment and (in) a social connectivity analysis scenario running on IoT farms.