A Generic Multi-node State Monitoring Subsystem

TL;DR

The paper presents a generic multi-node state monitoring subsystem that efficiently tracks and reports state changes across distributed systems without requiring knowledge of individual state machines, enhancing scalability and applicability.

Contribution

It introduces a novel, generic state monitoring approach that does not depend on specific state machine details, enabling broad application in multi-node systems.

Findings

Successfully monitors multiple nodes without knowledge of their internal states

Applied in BaBar DAQ system and a secondary system within the experiment

Provides real-time error detection and state coherence reporting

Abstract

The BaBar online data acquisition (DAQ) system includes approximately fifty Unix systems that collectively implement the level-three trigger. These systems all run the same code. Each of these systems has its own state, and this state is expected to change in response to changes in the overall DAQ system. A specialized subsystem has been developed to initiate processing on this collection of systems, and to monitor them both for error conditions and to ensure that they all follow the same state trajectory within a specifiable period of time. This subsystem receives start commands from the main DAQ run control system, and reports major coherent state changes, as well as error conditions, back to the run control system. This state monitoring subsystem has the novel feature that it does not know anything about the state machines that it is monitoring, and hence does not introduce any fundamentally new state machine into the overall system. This feature makes it trivially applicable to other multi-node subsystems. Indeed it has already found a second application beyond the level-three trigger, within the BaBar experiment.