Disk storage management for LHCb based on Data Popularity estimator

TL;DR

This paper introduces a machine learning-based algorithm for optimizing data storage in the LHCb system by predicting data popularity and adjusting dataset placement to save space and improve access times.

Contribution

It presents a novel machine learning approach for predicting data popularity and optimizing dataset placement in a hybrid storage system.

Findings

Significant disk space savings achieved.

Reduced job waiting times.

Effective prediction of data popularity.

Abstract

This paper presents an algorithm providing recommendations for optimizing the LHCb data storage. The LHCb data storage system is a hybrid system. All datasets are kept as archives on magnetic tapes. The most popular datasets are kept on disks. The algorithm takes the dataset usage history and metadata (size, type, configuration etc.) to generate a recommendation report. This article presents how we use machine learning algorithms to predict future data popularity. Using these predictions it is possible to estimate which datasets should be removed from disk. We use regression algorithms and time series analysis to find the optimal number of replicas for datasets that are kept on disk. Based on the data popularity and the number of replicas optimization, the algorithm minimizes a loss function to find the optimal data distribution. The loss function represents all requirements for data distribution in the data storage system. We demonstrate how our algorithm helps to save disk space and to reduce waiting times for jobs using this data.