A unified framework to improve the interoperability between HPC and Big Data languages and programming models

TL;DR

This paper introduces IgnisHPC, a unified framework that enables seamless execution and interoperability of HPC and Big Data workloads, supporting multi-language applications and leveraging MPI for performance improvements.

Contribution

The paper presents IgnisHPC, a novel framework unifying HPC and Big Data execution environments with native multi-language support and MPI-based communication.

Findings

IgnisHPC improves performance over Apache Spark.

It enables combining MPI and MapReduce tasks in the same framework.

The framework enhances productivity for HPC and Big Data applications.

Abstract

One of the most important issues in the path to the convergence of HPC and Big Data is caused by the differences in their software stacks. Despite some research efforts, the interoperability between their programming models and languages is still limited. To deal with this problem we introduce a new computing framework called IgnisHPC, whose main objective is to unify the execution of Big Data and HPC workloads in the same framework. IgnisHPC has native support for multi-language applications using JVM and non-JVM-based languages. Since MPI was used as its backbone technology, IgnisHPC takes advantage of many communication models and network architectures. Moreover, MPI applications can be directly executed in a efficient way in the framework. The main consequence is that users could combine in the same multi-language code HPC tasks (using MPI) with Big Data tasks (using MapReduce operations). The experimental evaluation demonstrates the benefits of our proposal in terms of performance and productivity with respect to other frameworks such as Apache Spark. IgnisHPC is publicly available for the Big Data and HPC research community.