A Redistribution Tool for Long-Term Archive of Astronomical Observation Data

TL;DR

This paper introduces AstroLayout, a tool that optimizes long-term astronomical data storage by redistributing data to improve access performance and reduce energy consumption across various storage media.

Contribution

It presents a novel graph partitioning algorithm for data redistribution, enhancing long-term archive efficiency for astronomical observation data.

Findings

Reduces the number of devices activated during data access

Improves data access performance in time-domain astronomy

Lowers power consumption of storage systems

Abstract

Astronomical observation data require long-term preservation, and the rapid accumulation of observation data makes it necessary to consider the cost of long-term archive storage. In addition to low-speed disk-based online storage, optical disk or tape-based offline storage can be used to save costs. However, for astronomical research that requires historical data (particularly time-domain astronomy), the performance and energy consumption of data-accessing techniques cause problems because the requested data (which are organized according to observation time) may be located across multiple storage devices. In this study, we design and develop a tool referred to as AstroLayout to redistribute the observation data using spatial aggregation. The core algorithm uses graph partitioning to generate an optimized data placement according to the original observation data statistics and the target storage system. For the given observation data, AstroLayout can copy the long-term archive in the target storage system in accordance with this placement. An efficiency evaluation shows that AstroLayout can reduce the number of devices activated when responding to data-access requests in time-domain astronomy research. In addition to improving the performance of data-accessing techniques, AstroLayout can also reduce the storage systems power consumption. For enhanced adaptability, it supports storage systems of any media, including optical disks, tapes, and hard disks.