Savu: A Python-based, MPI Framework for Simultaneous Processing of Multiple, N-dimensional, Large Tomography Datasets

TL;DR

Savu is a flexible Python-based MPI framework designed for efficient, parallel processing of large, multi-dimensional tomography datasets, facilitating high-throughput data analysis at synchrotron facilities.

Contribution

It introduces a modular, plugin-based processing pipeline that leverages parallel HDF5, enabling scalable and easy integration of new functionalities for large dataset processing.

Findings

Successfully deployed at DLS beamlines

Handles multiple n-dimensional datasets efficiently

Supports parallel processing across clusters

Abstract

Diamond Light Source (DLS), the UK synchrotron facility, attracts scientists from across the world to perform ground-breaking x-ray experiments. With over 3000 scientific users per year, vast amounts of data are collected across the experimental beamlines, with the highest volume of data collected during tomographic imaging experiments. A growing interest in tomography as an imaging technique, has led to an expansion in the range of experiments performed, in addition to a growth in the size of the data per experiment. Savu is a portable, flexible, scientific processing pipeline capable of processing multiple, n-dimensional datasets in serial on a PC, or in parallel across a cluster. Developed at DLS, and successfully deployed across the beamlines, it uses a modular plugin format to enable experiment-specific processing and utilises parallel HDF5 to remove RAM restrictions. The Savu design, described throughout this paper, focuses on easy integration of existing and new functionality, flexibility and ease of use for users and developers alike.