Abismal: Approximate Bayesian Inference for Scaling and Merging at Advanced Lightsources

TL;DR

Abismal is a new software that improves the processing of diffraction data using Bayesian inference and machine learning, making it more efficient for large-scale experiments.

Contribution

The novel contribution is an algorithmic innovation in Abismal that reduces memory requirements while maintaining Bayesian inference benefits.

Findings

Abismal addresses memory limitations in Bayesian inference for diffraction data processing.

The software supports high-throughput experiments with millions of diffraction images efficiently.

Abismal integrates with cctbx.xfel for user-friendly automated workflows during beam time.

Abstract

Experimental artifacts in diffraction data are typically corrected by an optimization algorithm known as scaling. After appropriate scaling, redundant measurements can then be merged. Recently, it was shown that unifying the two procedures, scaling and merging, into a single algorithm based on Bayesian (variational) inference and deep learning compared favorably to other programs for a broad range of experimental designs (Dalton et al., 2022). However, the memory requirements of this method pose challenges in serial crystallography and other data- intensive applications. In this talk, I will highlight the algorithmic innovation in the new software Abismal that addresses this limitation while preserving the benefits of the unifying Bayesian approach. Additionally, I will discuss ongoing efforts toward deploying and maintaining the software at LCLS and other light sources, including supporting a user-friendly interface with cctbx.xfel for automated workflows during beam time. By leveraging advances in machine learning, Abismal facilitates recovering structural signals in a robust and efficient way from high-throughput experiments with hundreds of thousands to millions of diffraction images.