Energy Profiling of Data-Sharing Pipelines: Modeling, Estimation, and Reuse Strategies

TL;DR

This paper presents a new modeling and estimation approach for energy consumption in data-sharing pipelines, highlighting reuse strategies to optimize energy efficiency across organizations.

Contribution

It introduces a novel method to model, estimate, and identify reuse opportunities for energy-efficient data-sharing pipelines.

Findings

Model accurately estimates energy consumption in pipelines

Reuse potential can significantly reduce energy use

Simulation shows promising optimization results

Abstract

Data-sharing pipelines involve a series of stages that apply policy-based data transformations to enable secure and effective data exchange among organizations. Although numerous tools and platforms exist to manage governance and enforcement in these pipelines, energy efficiency in data exchange has received limited attention. This paper introduces a novel method to model and estimate the energy consumption of different execution configurations in data-sharing pipelines. Additionally, this method identifies reuse potential in shared stages across pipelines that hold the key to reducing energy in large data-sharing federations. We validate this method through simulation experiments, revealing promising potential for cross-organizational pipeline optimization and laying a foundation for energy-conscious execution strategies.