A Modern Ethernet Data Acquisition Architecture for Fermilab Beam Instrumentation

TL;DR

This paper presents a modern Ethernet-based data acquisition architecture at Fermilab that replaces VME and VxWorks systems, increasing bandwidth and flexibility using open-source Linux and off-the-shelf Ethernet components.

Contribution

The paper introduces a scalable Ethernet-based data acquisition framework with a new protocol, replacing legacy hardware and enhancing bandwidth for Fermilab's beam instrumentation.

Findings

Replaced VME with Ethernet, reducing hardware complexity.

Achieved 10 Gbit/s total bandwidth using off-the-shelf Ethernet.

Successfully deployed across multiple Fermilab beamlines.

Abstract

The Fermilab Accelerator Division, Instrumentation Department is adopting an open-source framework to replace our embedded VME-based data acquisition systems. Utilizing an iterative methodology, we first moved to embedded Linux, removing the need for VxWorks. Next, we adopted Ethernet on each data acquisition module eliminating the need for the VME backplane in addition to communicating with a rack mount server. Development of DDCP (Distributed Data Communications Protocol), allowed for an abstraction between the firmware and software layers. Each data acquisition module was adapted to read out using 1 GbE and aggregated at a switch which up linked to a 10 GbE network. Current development includes scaling the system to aggregate more modules, to increase bandwidth to support multiple systems and to adopt MicroTCA as a crate technology. The architecture was utilized on various beamlines around the Fermilab complex including PIP2IT, FAST/IOTA and the Muon Delivery Ring. In summary, we were able to develop a data acquisition framework which incrementally replaces VxWorks & VME hardware as well as increases our total bandwidth to 10 Gbit/s using off the shelf Ethernet technology.