Using the Open Science Data Federation for data distribution: Big Bear Solar Observatory use case

TL;DR

The paper discusses the implementation of the Open Science Data Federation (OSDF) to enhance global data distribution, exemplified by its use with Big Bear Solar Observatory data for worldwide image processing pipelines.

Contribution

It introduces the expansion of OSDF based on StashCache, improving data accessibility, performance, and integration for scientific data sharing.

Findings

OSDF now has 20 origins and 30 caches, improving data distribution.

Integration with BBSO data enabled global access and processing pipelines.

OSDF has become vital to US cyber-infrastructure for scientific data sharing.

Abstract

The growing demand for extensive data processing is now a standard in many scientific fields. Efficiently distributing data to processing sites and enabling seamless sharing has become crucial. The Open Science Data Federation (OSDF) builds on the success of the StashCache project to establish a global data distribution network. By expanding StashCache, OSDF integrates additional data origins and caches, enhancing accessibility and performance (20 origins and 30 caches), new access methods, and monitoring and accounting mechanisms. Additionally, the OSDF has become essential to the US national cyber-infrastructure landscape due to the sharing requirements of recent NSF solicitations. One use case for the OSDF is the data access to the Big Bear Solar Observatory (BBSO). Integrating the BBSO data into the OSDF provided standard and reliable data access. Moreover, the OSDF caches provide local data worldwide. Using the OSDF and the BBSO data, creating a pipeline to apply image processing techniques to all images from BBSO anywhere on the planet was possible.