Towards Smart e-Infrastructures, A Community Driven Approach Based on Real Datasets

TL;DR

This paper introduces a community-driven, open source framework that uses machine learning to optimize resource utilization in data centers, aiming to improve efficiency and reduce energy consumption without sharing datasets.

Contribution

It presents a novel community-based framework leveraging machine learning for resource management in data centers, addressing data sharing and collaboration challenges.

Findings

Framework enables better resource utilization forecasting.

Reduces energy consumption in data centers.

Facilitates collaboration without dataset sharing.

Abstract

e-Infrastructures have powered the successful penetration of e-services across domains, and form the backbone of the modern computing landscape. e-Infrastructure is a broad term used for large, medium and small scale computing environments. The increasing sophistication and complexity of applications have led to even small-scale data centers consisting of thousands of interconnects. However, efficient utilization of resources in data centers remains a challenging task, mainly due to the complexity of managing physical nodes, network equipment, cooling systems, electricity, etc. This results in a very strong carbon footprint of this industry. In recent years, efforts based on machine learning approaches have shown promising results towards reducing energy consumption of data centers. Yet, practical solutions that can help data center operators in offering energy efficient services are lacking. This problem is more visible in the context of medium and small scale data center operators (the long tail of e-infrastructure providers). Additionally, a disconnect between solution providers (machine learning experts) and data center operators has been observed. This article presents a community-driven open source software framework that allows community members to develop better understanding of various aspects of resource utilization. The framework leverages machine learning models for forecasting and optimizing various parameters of data center operations, enabling improved efficiency, quality of service and lower energy consumption. Also, the proposed framework does not require datasets to be shared, which alleviates the extra effort of organizing, describing and anonymizing data in an appropriate format.