InstaCluster: Building A Big Data Cluster in Minutes

TL;DR

InstaCluster simplifies and accelerates the deployment of Big Data clusters on Amazon EC2, enabling users to set up complex environments in minutes without expert knowledge, thus reducing costs and improving reproducibility.

Contribution

The paper introduces InstaCluster, a tool that automates cluster setup on EC2, making Big Data infrastructure deployment quick, user-friendly, and reproducible.

Findings

Cluster deployment time reduced to minutes

No need for expert system administration skills

Cost-effective infrastructure provisioning

Abstract

Deploying, configuring, and managing large clusters is very a demanding and cumbersome task due to the complexity of such systems and the variety of skills needed. One needs to perform low-level configuration of the cluster nodes to ensure their interoperability and connectivity, as well as install, configure and provision the needed services. In this paper we address this problem and demonstrate how to build a Big Data analytic platform on Amazon EC2 in a matter of minutes. Moreover, to use our tool, embedded into a public Amazon Machine Image, the user does not need to be an expert in system administration or Big Data service configuration. Our tool dramatically reduces the time needed to provision clusters, as well as the cost of the infrastructure. Researchers enjoy an additional benefit of having a simple way to specify the experimental environments they use, so that their experiments can be easily reproduced by anyone using our tool.