TL;DR
HOSS is a flexible, Python-based system that enhances data handling for high-intensity experiments by parallelizing data writing and distributing data for online calibration, improving bandwidth and processing efficiency.
Contribution
It introduces a configurable, RDMA and zeroMQ-driven system for simultaneous data storage and processing in high-throughput experimental environments.
Findings
Improved data bandwidth for high-intensity experiments
Enables online data skimming and calibration
Uses RDMA, RAM disks, and zeroMQ for efficiency
Abstract
The Hall-D Online Skim System (HOSS) was developed to simultaneously solve two issues for the high intensity GlueX experiment. One was to parallelize the writing of raw data files to disk in order to improve bandwidth. The other was to distribute the raw data across multiple compute nodes in order to produce calibration \textit{skims} of the data online. The highly configurable system employs RDMA, RAM disks, and zeroMQ driven by Python to simultaneously store and process the full high intensity GlueX data stream.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.