D-SPACE4Cloud: A Design Tool for Big Data Applications

TL;DR

This paper introduces D-SPACE4Cloud, a tool that optimizes cloud cluster configurations for big data applications to minimize costs while meeting quality of service requirements.

Contribution

It presents a novel integrated optimization and prediction tool specifically designed for cloud-based big data system configuration.

Findings

Validated on real systems, demonstrating effectiveness.

Reduces deployment costs while satisfying QoS constraints.

Provides a systematic approach for hardware configuration design.

Abstract

The last years have seen a steep rise in data generation worldwide, with the development and widespread adoption of several software projects targeting the Big Data paradigm. Many companies currently engage in Big Data analytics as part of their core business activities, nonetheless there are no tools and techniques to support the design of the underlying hardware configuration backing such systems. In particular, the focus in this report is set on Cloud deployed clusters, which represent a cost-effective alternative to on premises installations. We propose a novel tool implementing a battery of optimization and prediction techniques integrated so as to efficiently assess several alternative resource configurations, in order to determine the minimum cost cluster deployment satisfying QoS constraints. Further, the experimental campaign conducted on real systems shows the validity and relevance of the proposed method.