The Design of a Community Science Cloud: The Open Science Data Cloud Perspective

TL;DR

This paper details the design and implementation of the Open Science Data Cloud, a large-scale infrastructure supporting scientists with petabyte-scale data storage and processing capabilities across multiple disciplines.

Contribution

It introduces the architecture and operational experience of a petabyte-scale data cloud infrastructure for scientific research, enabling diverse scientific projects.

Findings

Successfully deployed over 2000 cores and 2 PB storage across four data centers.

Enabled multiple research projects in biology, earth sciences, and social sciences.

Shared lessons learned and software stack details from three years of operation.

Abstract

In this paper we describe the design, and implementation of the Open Science Data Cloud, or OSDC. The goal of the OSDC is to provide petabyte-scale data cloud infrastructure and related services for scientists working with large quantities of data. Currently, the OSDC consists of more than 2000 cores and 2 PB of storage distributed across four data centers connected by 10G networks. We discuss some of the lessons learned during the past three years of operation and describe the software stacks used in the OSDC. We also describe some of the research projects in biology, the earth sciences, and social sciences enabled by the OSDC.