SDN helps Big Data to optimize access to data

TL;DR

This paper explores how Software-Defined Networking (SDN) can optimize data access in Big Data and HPC integration, reducing transfer times and improving overall application performance through innovative solutions and a new framework.

Contribution

It introduces a novel SDN-based approach and the DataBroker framework for enhancing data transfer efficiency in HPC and Big Data environments.

Findings

SDN reduces data transfer times in HPC-Big Data systems.

The DataBroker framework improves data access performance.

Experimental results validate the effectiveness of the proposed solutions.

Abstract

This chapter introduces the state-of-the-art in the emerging area of combining High Performance Computing (HPC) with Big Data Analysis. To understand the new area, the chapter first surveys the existing approaches to integrating HPC with Big Data. Next, the chapter introduces several optimization solutions that focus on how to minimize the data transfer time from computation-intensive applications to analysis-intensive applications as well as minimizing the end-to-end time-to-solution. The solutions utilize SDN to adaptively use both high speed interconnect network and high performance parallel file systems to optimize the application performance. A computational framework called DataBroker is designed and developed to enable a tight integration of HPC with data analysis. Multiple types of experiments have been conducted to show different performance issues in both message passing and parallel file systems and to verify the effectiveness of the proposed research approaches.