In-situ Workflow Auto-tuning via Combining Performance Models of Component Applications

TL;DR

This paper introduces CEAL, an auto-tuning method for in-situ workflows that efficiently optimizes performance by combining machine learning with workflow structure knowledge, reducing the need for extensive measurements.

Contribution

The paper presents CEAL, a novel auto-tuning approach that leverages component interactions and active learning to efficiently optimize in-situ workflow configurations.

Findings

CEAL reduces the number of performance measurements needed for tuning.

It achieves near-optimal configurations with limited training data.

Demonstrates effectiveness on real in-situ workflow scenarios.

Abstract

In-situ parallel workflows couple multiple component applications, such as simulation and analysis, via streaming data transfer. in order to avoid data exchange via shared file systems. Such workflows are challenging to configure for optimal performance due to the large space of possible configurations. Expert experience is rarely sufficient to identify optimal configurations, and existing empirical auto-tuning approaches are inefficient due to the high cost of obtaining training data for machine learning models. It is also infeasible to optimize individual components independently, due to component interactions. We propose here a new auto-tuning method, Component-based Ensemble Active Learning (CEAL), that combines machine learning techniques with knowledge of in-situ workflow structure to enable automated workflow configuration with a limited number of performance measurements.