Sea: A lightweight data-placement library for Big Data scientific computing

TL;DR

Sea is a lightweight library that enables data-placement strategies for scientific applications on HPC clusters without requiring workflow reinstrumentation, significantly improving performance in data-intensive tasks.

Contribution

It introduces Sea, a novel library leveraging GNU C interception to facilitate data placement in scientific workflows without reinstrumentation.

Findings

Sea achieves up to 3x performance improvement.

It effectively integrates with existing scientific applications.

Demonstrated on neuroimaging data with synthetic benchmarks.

Abstract

The recent influx of open scientific data has contributed to the transitioning of scientific computing from compute intensive to data intensive. Whereas many Big Data frameworks exist that minimize the cost of data transfers, few scientific applications integrate these frameworks or adopt data-placement strategies to mitigate the costs. Scientific applications commonly rely on well-established command-line tools that would require complete reinstrumentation in order to incorporate existing frameworks. We developed Sea as a means to enable data-placement strategies for scientific applications executing on HPC clusters without the need to reinstrument workflows. Sea leverages GNU C library interception to intercept POSIX-compliant file system calls made by the applications. We designed a performance model and evaluated the performance of Sea on a synthetic data-intensive application processing a representative neuroimaging dataset (the Big Brain). Our results demonstrate that Sea significantly improves performance, up to a factor of 3$\times$.