Information Systems Playground - The Target Infrastructure, Scaling Astro-WISE into the Petabyte range

TL;DR

This paper describes the development of a scalable, high-capacity storage and compute infrastructure for Astro-WISE applications within the Target project, supporting petabyte-scale data management and processing.

Contribution

It introduces a tailored infrastructure with 10 PB storage and thousands of cores, utilizing IBM's Global Parallel File System for scalability and flexibility.

Findings

Achieved 10 PB storage capacity for Astro-WISE applications.

Enabled scalable compute resources with thousands of cores.

Demonstrated flexible, future-proof storage architecture.

Abstract

The Target infrastructure has been specially built as a storage and compute infrastructure for the information systems derived from Astro-WISE. This infrastructure will be used by several applications that collaborate in the area of information systems within the Target project. It currently consists of 10 PB of storage and thousands of computational cores. The infrastructure has been constructed based on the requirements of the applications. The storage is controlled by the Global Parallel File System of IBM. This file system takes care of the required flexibility by combining storage hardware with different characteristics into a single file system. It is also very scalable, which allows the system to be extended into the future, while replacing old hardware with new technology.