Bringing computation to the data: A MOEA-driven approach for optimising data processing in the context of the SKA and SRCNet

TL;DR

This paper presents a MOEA-driven framework for optimizing data processing workflows by moving computation closer to data sources, addressing the challenges of large-scale data in SKA and SRCNet environments.

Contribution

It introduces a novel integration of FaaS with MOEAs to optimize execution plans considering time, energy, and data transfer costs in distributed data processing.

Findings

MOEAs effectively optimize execution plans balancing multiple objectives.

The framework reduces data transfer and energy consumption in data-intensive workflows.

A baseline for cost-aware computation-to-data strategies is established.

Abstract

The Square Kilometre Array (SKA) will generate unprecedented data volumes, making efficient data processing a critical challenge. Within this context, the SKA Regional Centres Network (SRCNet) must operate in a near-exascale environment where traditional data-centric computing models based on moving large datasets to centralised resources are no longer viable due to network and storage bottlenecks. To address this limitation, this work proposes a shift towards distributed and in-situ computing, where computation is moved closer to the data. We explore the integration of Function-as-a-Service (FaaS) with an intelligent decision-making entity based on Evolutionary Algorithms (EAs) to optimise data-intensive workflows within SRCNet. FaaS enables lightweight and modular function execution near data sources while abstracting infrastructure management. The proposed decision-making entity employs Multi-Objective Evolutionary Algorithms (MOEAs) to explore near-optimal execution plans considering execution time and energy consumption, together with constraints related to data location and transfer costs. This work establishes a baseline framework for efficient and cost-aware computation-to-data strategies within the SRCNet architecture.