Homomorphic data compression for real time photon correlation analysis

TL;DR

This paper introduces a homomorphic compression method that significantly accelerates real-time photon correlation analysis by enabling direct operations on compressed data, addressing big data challenges in x-ray science.

Contribution

The authors develop a novel homomorphic compression scheme that allows direct correlation calculations on compressed data, reducing computational time by 10,000 times for XPCS.

Findings

Achieved real-time correlation analysis at kHz framerate.

Reduced computational time by a factor of 10,000.

Demonstrated effective big data management at x-ray facilities.

Abstract

The construction of highly coherent x-ray sources has enabled new research opportunities across the scientific landscape. The maximum raw data rate per beamline now exceeds 40 GB/s, posing unprecedented challenges for the online processing and offline storage of the big data. Such challenge is particularly prominent for x-ray photon correlation spectroscopy (XPCS), where real time analyses require simultaneous calculation on all the previously acquired data in the time series. We present a homomorphic compression scheme to effectively reduce the computational time and memory space required for XPCS analysis. Leveraging similarities in the mathematical expression between a matrix-based compression algorithm and the correlation calculation, our approach allows direct operation on the compressed data without their decompression. The lossy compression reduces the computational time by a factor of 10,000, enabling real time calculation of the correlation functions at kHz framerate. Our demonstration of a homomorphic compression of scientific data provides an effective solution to the big data challenge at coherent light sources. Beyond the example shown in this work, the framework can be extended to facilitate real-time operations directly on a compressed data stream for other techniques.