S2CE: A Hybrid Cloud and Edge Orchestrator for Mining Exascale Distributed Streams

TL;DR

This paper introduces S2CE, a scalable hybrid cloud and edge orchestrator designed to efficiently process and analyze exascale, heterogeneous data streams from distributed IoT devices using machine learning techniques.

Contribution

It presents the first optimized multi-cloud and edge orchestrator capable of scalable, configurable data processing for exascale distributed streams, integrating data fusion, resource management, and distributed processing.

Findings

Enables machine learning on large heterogeneous data streams.

Supports scalable and configurable hybrid cloud-edge processing.

Facilitates practical data fusion and resource management.

Abstract

The explosive increase in volume, velocity, variety, and veracity of data generated by distributed and heterogeneous nodes such as IoT and other devices, continuously challenge the state of art in big data processing platforms and mining techniques. Consequently, it reveals an urgent need to address the ever-growing gap between this expected exascale data generation and the extraction of insights from these data. To address this need, this paper proposes Stream to Cloud & Edge (S2CE), a first of its kind, optimized, multi-cloud and edge orchestrator, easily configurable, scalable, and extensible. S2CE will enable machine and deep learning over voluminous and heterogeneous data streams running on hybrid cloud and edge settings, while offering the necessary functionalities for practical and scalable processing: data fusion and preprocessing, sampling and synthetic stream generation, cloud and edge smart resource management, and distributed processing.