Reconstruction and Analysis on Demand: A Success Story

TL;DR

This paper presents a lazy evaluation-based reconstruction system for high-energy physics that improves efficiency, configurability, and robustness, leading to widespread adoption by physicists.

Contribution

It introduces a novel on-demand processing approach with Producers and Processors, enhancing flexibility and efficiency over traditional linear module sequences.

Findings

Proven four-year successful use in CLEO III system

Significant improvements in data access ease and efficiency

Enhanced event filtering and data correctness guarantees

Abstract

The traditional design of an HEP reconstruction system partitions the problem into a series of modules. A reconstruction job is then just a sequence of modules run in a particular order with each module reading data from the event and placing new data into the event. The problem with such a design is it is up to the user to place the modules in the correct order and CPU time is wasted calculating quantities that may not be used if the event is rejected based on some other criteria. The CLEO III analysis/reconstruction system takes a different approach: on demand processing (otherwise known as lazy evaluation). Jobs are still partitioned into smaller components which we call Producers. However, Producers register what data they produce. The first time a datum is requested for an event the Producer's algorithm is run. Sources work similarly, registering what data they can retrieve but delaying retrieval until the data is requested. Data analysis and filtering are done via a separate set of modules called Processors. We have been using this system for four years and it has been a huge success. The data access implementation has proven to be very easy to use and extremely efficient. Reconstruction jobs are easy to configure and additional event filters can be added efficiently and robustly -- access to correct data is automatically guaranteed. The true test of success: physicists have embraced this model for analysis even though the old model is still supported.