The XENON1T Data Acquisition System

TL;DR

The XENON1T data acquisition system is a highly sensitive, low-threshold, and high-bandwidth setup that efficiently digitizes and stores signals from the detector, enabling three years of successful operation in dark matter research.

Contribution

This paper presents a novel DAQ system combining commercial, open source, and custom components with advanced trigger and database techniques for high-efficiency detection in dark matter experiments.

Findings

Achieved sub-0.1 photoelectron threshold

Over 97% event recognition efficiency

Supported over 300 MB/s bandwidth during calibration

Abstract

The XENON1T liquid xenon time projection chamber is the most sensitive detector built to date for the measurement of direct interactions of weakly interacting massive particles with normal matter. The data acquisition system (DAQ) is constructed from commercial, open source, and custom components to digitize signals from the detector and store them for later analysis. The system achieves an extremely low signal threshold below a tenth of a photoelectron using a parallelized readout with the global trigger deferred to a later, software stage. The event identification is based on MongoDB database queries and has over 97% efficiency at recognizing interactions at the analysis energy threshold. A readout bandwidth over 300 MB/s is reached in calibration modes and is further expandable via parallelization. This DAQ system was successfully used during three years of operation of XENON1T.

Tables

Table 1. Table 1 : Event rates and sizes for various operating modes in XENON1T. For the internal sources \ce ​ R 220 ​ n \ce superscript 𝑅 220 𝑛 \ce{{}^{220}Rn} and \ce ​ K 83 ​ m ​ r \ce superscript 𝐾 83 𝑚 𝑟 \ce{{}^{83m}Kr} the event rate is highest right after the source is injected and decreases to the background rate over the course of several hours, at which point either another injection is performed or the calibration campaign is ended. The numbers with an asterisk indicate the event size after application of the hardware high energy veto, described in Section 2.1 .

Operation Mode	Maximum Event Rate (Hz)	Average Event Size (MB)
Dark matter search mode	6	1.68
${}_{}{}^{220}\mathrm{Rn}$	100	1.66 (0.77*)
${}_{}{}^{83m}\mathrm{Kr}$	200	0.33
${}_{}{}^{241}\mathrm{AmBe}$	11	2.19
Neutron Generator	30	2.84 (1.02*)
LED	450	0.30

Equations

$$\mathrm{Event}\mathrm{Size}=10\mathrm{kB}+4\frac{\mathrm{B}}{\mathrm{PE}}\times S2$$

Full text

1 Overview

The XENON1T experiment is the largest dual-phase liquid xenon time projection chamber (TPC) built to date. It was designed to measure nuclear recoils induced by dark matter in the form of weakly interacting massive particles (WIMPs). The detector has been in continuous operation at the INFN Laboratori Nazionali del Gran Sasso (LNGS) in Italy from its commissioning in 2016 until December 2018 and has produced the strongest exclusion limits to date on the coupling of WIMPs to matter [1, 2, 3]. Particles incident on the 2.0 tonne liquid xenon target produce a prompt scintillation signal (S1) and delayed ionization signal, measured via proportional scintillation (S2), separated by a maximum time delay of 670 $\upmu$s. These signals are measured by two arrays of 127 and 121 Hamamatsu R11410-21 3" photomultiplier tubes (PMTs) [5] installed above and below the liquid xenon target. The energy region of interest for the standard spin-independent WIMP search is $[4.9-40.9]~{}\mathrm{keV}$ for nuclear recoils, but the bulk of background events, as well as regions of interest for other physics searches [4], are on scales up to a few MeV. The detector is installed in a large tank of deionized water instrumented as an active Cherenkov muon veto in order to reduce background and identify cosmic muons [6]. The XENON1T detector is described in detail in [7].

The data acquisition system (DAQ) is responsible for digitizing the signals from each PMT and identifying and recording every interaction with the target. The large variation in signal intensity (keV up to MeV scale), as well as the two very different signal shapes of the S1 and S2 and the variable time delay between them, create a unique challenge for the DAQ. The DAQ is designed around a triggerless readout paradigm, where data is acquired constantly without a global trigger and physical interactions are identified later in software ("event building"). This allows for the lowest possible energy threshold, but necessitates custom readout logic to provide independent readout of each detector channel while maintaining full time synchronization. The system is centered around a NoSQL MongoDB database [8], which provides fast storage, sorting, and retrieval of data. A software event builder queries this database asynchronously to identify interactions and write them to files. During normal operation the event builder runs within minutes of acquisition, but can be delayed for hours or days in case of disruptions further down the pipeline. The files are transferred to external sites and processed into a format useful for analysis by the XENON1T computing system [9, 10]. A diagram of the DAQ is shown in Figure 1 and the subsystems are detailed in the following sections.

In dark matter search mode, the event rate identified by the event builder is about 5 Hz and the readout runs with <1% deadtime, which is caused by long, periodic electrical noise emitted intermittently by individual channels. Nuclear recoil calibrations, which measure the detector response to a WIMP-like signal, are performed with an $\ce{{}^{241}AmBe}$ source or a deuterium-deuterium neutron generator [11] placed in the water tank outside the cryostat and produce event rates of 10 Hz and 25 Hz, respectively. Other calibrations utilize radioactive sources mixed into the xenon target itself. $\ce{{}^{83m}Kr}$ is injected once every few weeks for detector stability monitoring [12]. $\ce{{}^{220}Rn}$ is used to measure the response to background-like low energy electronic recoil calibration events and is injected periodically throughout the science run [13]. $\ce{{}^{83m}Kr}$ and $\ce{{}^{220}Rn}$ calibrations have maximum event rates of 200 Hz and 100 Hz, respectively. Because XENON1T is a rare event search, the full digitized waveform of every event must be recorded for later scrutiny, necessitating a high throughput capability in calibration modes. In addition to these calibrations using radioactive sources, the PMT gains are calibrated with pulsed LEDs configured to provide signals at the single photoelectron level [5]. Table 1 shows an overview of the various operating modes as well as their maximum event rates and average event sizes.

2 Hardware and Readout

The readout portion of the DAQ encompasses all components up to the MongoDB buffer in Figure 1. This stage is responsible for detecting signals from the TPC and putting them into a digital format suitable for software triggering.

2.1 Frontend Electronics

The electronics of XENON1T are housed in a counting room within a service building adjacent to the water tank containing the detector [7]. PMT high voltage is controlled via the XENON1T slow control system [14] and supplied by CAEN A1535N modules connected via custom-made passive low-pass filter boxes with a 2.8 MHz cutoff frequency to reduce electronic noise. The filter boxes reduce the RMS of the baseline from a mean of $(0.91\pm 0.19)$ mV to $(0.31\pm 0.06)$ mV, with the remaining dominant 25 kHz noise frequency originating from the high voltage power supplies themselves. Photons incident on the PMT photocathode are detected with an average quantum efficiency of 34.5% and amplified by a channel-dependent gain of $(1.0-5.0)\times 10^{6}~{}$[5]. The signals are routed by $12$ m PTFE-coated RG-196 coaxial cables [5] to the counting room, passing through potted vacuum feedthroughs (RHSeals), and connected to Phillips model 776 amplifiers [15]. The amplifiers provide a fixed DC-coupled, non-inverting voltage gain of 10. One of the two outputs is connected to one of eight input channels on a CAEN model V1724 digitizer module in order to record signals from each PMT [16]. 31 of these modules are operated synchronously to record data from the TPC and one additional module records diagnostic data from the DAQ itself. One V1724 provides a 50 MHz clock synchronization signal that is propagated to all of the others such that the clock drift is constant across all digitizers. Acquisition start is synchronized with a NIM logic signal sent from a CAEN V2718 crate controller in order to start the clock counters on all digitizers at the same instant. The V1724s digitize the input signals at a rate of 100 MHz with 2.25 V input range, 40 MHz input bandwidth, and 14 bit resolution. Data is read out from the digitizers by server-grade PCs via 7 optical links connected to CAEN A3818 PCIe cards with either 4 or 8 daisy-chained digitizers per link. Each optical link supports a readout bandwidth of up to 90 MB/s. By parallelizing readout over multiple optical links and PCs, a maximum readout rate of 300 MB/s is achieved for the complete system, which is sufficient to record the 100 Hz target rate for the $\ce{{}^{220}Rn}$ calibration mode. The maximum readout bandwidth is defined by the saturation point of the first optical link to reach its peak bandwidth of 90 MB/s, since beyond that point the DAQ will be capable of recording only partial events. Above about 100 Hz in $\ce{{}^{220}Rn}$ data, events begin to overlap due to long periods of activity following large S2 signals, during which many single electrons are extracted into the gas region [17].

Except during LED calibration measurements, which utilize an external periodic NIM trigger provided by a CAEN V2718 crate controller module, digitizers are operated using a custom self-triggering firmware developed together with CAEN as a modification of their DPP-DAW firmware. It implements an independent threshold trigger on each channel ("self-trigger"), which stores and reads out the sampled PMT waveform of a single channel any time a signal on that channel exceeds a configurable threshold. The empty baseline data outside of the self-trigger window is not recorded. The firmware allows user configuration of the self-trigger threshold and the pre- and post-trigger acquisition windows. These windows are each set to 50 samples, corresponding to 500 ns, in order to provide enough empty baseline around each pulse for accurate fitting and subtraction. In case signals exceed the configured acquisition length, the window is automatically extended until the signal goes under threshold again for a user-configurable number of samples, allowing deadtime-less acquisition of both short, small S1 pulses and much larger and longer S2 pulses. This behavior is demonstrated in Figure 2, which shows two self-triggered pulses. The pulse on the left axis is a single photoelectron (PE) and is the shortest length pulse possible (1 $\upmu$s), while the pulse on the right is a larger S2 that is recorded in its entirety by dynamically extending the acquisition window, in this case to more than 8 $\upmu$s. The maximum self-trigger length is defined by the size of the digitizers’ internal memory and is over 3 ms. This is orders of magnitude longer than the longest pulses found in normal readout conditions during a XENON1T science run. The self-trigger threshold was set to 15 ADC counts, or 2.06 mV for the majority of channels, but is raised to a maximum of 30 ADC counts for 30 channels exhibiting abnormally high self-trigger rates due to periodic electronic noise from the high voltage power supplies or abnormal PMT behavior. The channel-dependent single PE acceptance averages to $(93\pm 3)\%$ [1].

2.2 Readout Software

Readout PCs run the kodiaq software package [18], which reformats data from the binary CAEN format to MongoDB documents. Data is transferred from the digitizers to the readout PCs via block transfers mediated by the CAENVMElib C++ library. The readout software constantly polls the boards in round-robin fashion to read out any available new data. Upon completion of a block transfer, the transferred data is parsed into individual pulses. A pulse refers to a single self-trigger from a single channel, which, together with its identifying meta-data, constitutes a single entry in the buffer database. During acquisition one software thread is dedicated to constantly polling the boards for new data in order to maximize readout bandwidth, while several other threads work to reformat the data as it arrives. MongoDB is a NoSQL database which organizes data into collections, analogous to tables in an SQL database. Single database entries in MongoDB are schema-less and are called documents (similar to SQL rows). Each document written by kodiaq contains the timestamp since run start in units of nanoseconds with 10 ns resolution, digitizer module number, channel number, compressed waveform samples as a binary field, and a MongoDB ObjectID field as a unique identifier. Kodiaq clients on different physical hosts (readout PCs) are controlled by a single dispatcher software module, which interfaces with the user interface described later. The dispatcher monitors the state of the readout nodes, collects diagnostic information, sends commands, and creates and indexes collections in the buffer databases, which store data that is waiting to be triggered. In case one or more readout processes suffer a malfunction, the dispatcher stops the entire DAQ and propagates the error message to the user interface as well as to the XENON1T slow control system, where an alarm may be raised.

The dispatcher is also responsible for interfacing with the runs database (see figure 1) and creating metadata records for each run (see section 4.1). The dispatcher/slave architecture facilitates parallelization across several physical readout servers, enabling the system to reach a peak readout bandwidth of 300 MB/s. Communication between the dispatcher and slaves is performed via a custom protocol running over network sockets within the DAQ network. There is no hard limit on the level of parallelization possible and the system could in principle easily be extended by adding more hardware, with the eventual limit coming at the event building stage where all the independent readout pipelines must be combined by one process (see Section 3).

2.3 Busy and High-Energy Veto Systems

The CAEN V1724 digitizers have 8 MB onboard dual-ported memory per channel and will populate this buffer until the data are read by the readout software, at which point the memory is made available again. If the incoming data fills the buffer faster than it can be cleared, this memory can fill completely and the digitizer stops storing new events. In this case a busy signal is raised by the digitizer on its front panel LVDS interface. The busy signals from all digitizers are propagated to a CAEN V1495 general purpose module where a custom FPGA firmware is configured to send a NIM logic signal in case any of the 31 V1724 data-transferring boards report a memory full condition. The NIM signal is fanned to an input on each board and the presence of this signal inhibits the board from recording new data. Therefore, if one board goes into the busy state, the data stream for all boards is effectively blocked until this state is lifted. This busy state is monitored via an additional digitizer, the acquisition monitor, described in Section 2.4. When the DAQ is operating using a hardware trigger, for example during LED calibrations and noise monitoring, the trigger NIM logic signal is passed through the V1495 module and inhibited in case any single board reports a busy state. This scheme ensures that for every recorded event, all data from all active channels is available. Partial events during which a busy condition was raised are removed from the later analysis using data from the acquisition monitor.

During neutron generator and ${}^{220}\mathrm{Rn}$ calibration modes, the size of the data is greatly increased by high energy events outside the region of interest for low energy calibrations. In the case of ${}^{220}\mathrm{Rn}$ these are high energy $\alpha$-decay events and in the case of neutron generator data these are typically $\gamma$-decays following neutron capture processes. These high energy events are followed by long tails of delayed single electrons leading to a very high storage and bandwidth footprint and can cause unacceptably high dead-times, despite not being useful for most analyses. The (compressed) storage size in bytes of a XENON1T event is S2 dependent and is determined by data to be approximately

[TABLE]

Here S2 represents the size of the S2 in PE. The constant term can be interpreted as the storage size required for digitizing an S1 and S2 plus the storage overhead, while the S2-dependent term represents the contribution of the post-S2 tail. There are multiple orders of magnitude in size difference between a typical low energy event ($\mathrm{S2}\sim 10^{2}-10^{3}~{}\mathrm{PE}$) and a typical $\alpha$-decay event in $\ce{{}^{220}Rn}$ calibration data ($\mathrm{S2}~{}>~{}10^{5}~{}\mathrm{PE}$). In order to suppress these high energy events and preserve readout bandwidth for pertinent low-energy data, a veto system was developed based on a Skutek digital pulse processor DDC-10. It uses a BlackVME S6 motherboard hosting a Blackfin BF561 processor, programmable Spartan-6 LX150 FPGA, and a 10 channel 14-bit ADC card sampling at 100 MHz [19]. The input to this unit is the analog sum of the bottom array PMT channels, which is created using the second output of the Phillips 776 amplifiers and a series of Phillips 740 linear fan-in modules, the last two of which are modified with 30 dB attenuators to adjust the signal to be within the 2 V dynamic range of the DDC-10 digitizer input. A custom FPGA firmware on the high energy veto module constantly scans the input signal and searches for pulses with shapes compatible with S2s and integrated areas above a user-configurable threshold. This is set to approximately $1.5\times 10^{5}$ PE, corresponding to several hundred keV. When a signal meeting these requirements is found, a NIM logic signal with a fixed length of 10 ms is generated and propagated to the digitizers using the same fan array used by the busy veto in order to inhibit acquisition of both the high-energy S2 and the most intense component of the subsequent single-electron tail. The storage size for one hour of ${}^{220}\mathrm{Rn}$ calibration data is about $150~{}\mathrm{GB}$ without or $35~{}\mathrm{GB}$ with the high energy veto active with no impact on the low energy calibration signals. Figure 3 illustrates the functionality of the high energy veto in an example event.

Both the high-energy and busy veto systems take advantage of another feature of the XENON custom CAEN V1724 digitizer firmware where data is not processed by the internal logic immediately but delayed at the input stage in a FIFO by a user configurable time of up to 10.22 $\upmu$s (XENON1T uses this maximum value). Thus, a high energy S2 inducing a high-energy veto signal will be removed from the data stream by the veto signal that it prompted. This can be seen in Figure 3, in which the veto signal is received in the data stream (red line) before it is actually issued in real time (purple line).

2.4 Acquisition Monitor

In addition to the 31 V1724 modules used to digitize the signals from the TPC PMTs, one additional V1724 module, the acquisition monitor, is used to digitize information about the DAQ system itself. This module is not connected to the busy module and is fed with a low enough rate of signals to ensure that it is never busy. In order to allow each channel to function independently, this digitizer is also equipped with the XENON custom firmware and operated in self-triggering mode. One channel records the attenuated analog sum of the bottom-array PMTs that is used for the high-energy veto input. If the DAQ does go busy due to high rate or a high-energy veto signal there is still a reduced level of information recorded from the detector during this time, as can be seen when comparing the top (sum signal recorded by the acquisition monitor) versus bottom (full resolution data) panels of Figure 3. The other channels record the busy on/off, the high energy veto on/off, and the trigger from the muon veto system, which is used as an additional synchronization for the two systems (see section 2.5). The acquisition monitor data is stored in the main data stream and is available to analysts in high level data. The first level of selection conditions in the XENON1T analysis, which remove events where the busy, high-energy, or muon veto systems were active, relies on this data [20].

2.5 Muon Veto Detector Readout

The water-Cherenkov muon veto detector surrounding the TPC consists of 84 Hamamatsu R5912-HQE PMTs, read out with a second DAQ system that uses a coincidence trigger module to induce acquisition in 11 CAEN V1724 digitizers via a hardware trigger [6]. Besides the different triggering paradigm, this system is nearly identical to the TPC system described in detail here and both systems share the same readout software, data flow framework, and analysis software. The data from the two detectors is combined at the analysis stage in order to remove events originating from a muon shower, especially muon-induced nuclear recoils.

The 50 MHz clock synchronization signal used by the TPC digitizers is also propagated to the muon veto modules to keep the clocks for both readout systems synchronized. In addition, a GPS-based absolute timing module is embedded into the muon veto system to provide time synchronization with potential external phenomena, such as galactic supernovae [21]. A receiver module located on the laboratory surface computes the current time according to signals from GPS satellites and is connected by a fiber optic cable to a second, underground module located in the XENON counting room. Each time the muon veto is triggered, this module records and stores the current GPS time. In addition, a NIM logic signal is emitted every 15 seconds, which triggers the GPS module and is simultaneously recorded by the acquisition monitor. To compute the exact, global time for a given event, the DAQ retrieves the GPS time of the most recent muon veto or synchronization trigger and adds the time difference according to the digitizer clock. There is at most one bin (10 ns) of jitter over these few seconds, which is negligible for the purposes of this measurement. The DAQ for the upcoming XENONnT experiment will continue to use this method for time synchronization.

3 Software Event Builder

The continuous stream of data produced by the readout must be segmented into individual events for use in XENON1T’s data processor PAX [9]. The data in between events, consisting primarily of PMT dark counts and single electrons trailing high-energy events, is discarded to save storage space, except in special run modes for studying low-threshold, high-background processes.

The software trigger and event builder [22] perform this segmenting and pruning of the data stream. They read from the MongoDB buffer, search for events compatible with interactions in the detector, and produce archive files with events that are ready for further processing.

3.1 MongoDB Buffer Databases

The readout software, kodiaq, inserts data into three standalone MongoDB databases configured on three event builder computers. MongoDB’s “sharding" feature would have allowed operation of a single multi-host database, but it proved too slow (in various configurations) to maintain real-time insertion and query rates in calibration modes, likely due to overhead in maintenance of the combined index. Instead, the readout is configured such that a roughly equal amount of data is inserted into each of the three standalone databases by assigning one third of the digitizer modules to write to each database. In order to work around some aspects of the MongoDB internal memory management, most notably that the deletion of a document does not automatically free up the corresponding disk space, a new MongoDB collection is created every  21 seconds on each machine and the data load is transfered to the newest collection. When the data in the collection is no longer needed, the entire collection is dropped, which deletes all data and indexes and frees up the corresponding storage space. The 21-second time window corresponds to the wraparound of the 31-bit, 100 MHz V1724 digitizer clock counter. All collections are indexed at initialization by timestamp, module identifier, and channel number, allowing for fast queries at the event builder stage. The trigger logic is able to detect pulses that span two collections by postponing processing of pulses appearing near the end of one collection until the next collection is read.

The database on each machine can use up to 128 GB RAM, 1 TB solid state storage, and several TB hard disk space, allowing the DAQ to record data for days even if the connection to the file storage system is interrupted. During calibration data-taking, data is cleared by dropping each 21-second collection immediately after it is processed by the trigger, which typically happens less than two minutes after acquisition. During dark matter data-taking, pre-trigger data is stored in the buffer databases for six hours before being cleared. An external program monitors the Supernova Early Warning System (SNEWS) mail list and only flags data for purging if there is no supernova email warning [23]. This would enable the study of low-energy signals from supernova neutrinos, which may otherwise be partially below the trigger threshold [24].

3.2 Trigger Logic and Performance

The XENON1T software trigger logic is a single thread process included as part of the PAX data processor [9]. It uses a rolling time window to query the MongoDB database and search for time-coincident pulses. Unpacking the full data payload of each pulse and combining this information would require a system with a prohibitively large amount of computing power, especially during high rate calibrations. Therefore, the payload (digitized waveform) of each self-trigger document is never examined, only the timestamp and channel number. These fields are indexed by the database, allowing the trigger to function in real time. The trigger logic is able to effectively identify S1 and S2 signals by recognizing the tight time coincidence (within 100 ns) of S1 signals compared to the broader distribution of S2 signals, which can be several $\upmu\mathrm{s}$ long. After classification as S1 or S2, a trigger decision is made based on the number of pulses that contribute to the signal, with the coincidence level set to 50 pulses for S1 and 60 pulses for S2.

The low energy threshold of the XENON1T spin-(in)dependent WIMP search analyses [1, 2, 3] is defined by a 3-fold coincidence requirement on the S1 signal. Due to the amplification of the S2 signal, interactions with S1s at this energy feature S2s that are orders of magnitude higher. Thus, it is the S2 trigger channel that is relevant for the low-energy performance of the DAQ. The threshold of 60 pulses eliminates triggers from one- or two-electron S2s, which are common following high-energy events, while achieving a high efficiency at the target analysis S2 threshold of 200 PE. Lower-energy signals can still be studied in special run modes where the trigger is disabled. Due to larger instrumental backgrounds at these low energies, only a small amount of such data is needed to get background-limited studies.

Figure 4 shows the trigger efficiency as a function of S2 size, as determined from simulated waveforms (curves) and measured using $\ce{{}^{220}Rn}$ calibration data events near the cathode (points). Due to large charge losses at the cathode (caused by the field inversion), events originating there can have S2s around the trigger threshold even if their S1s are large enough to induce a trigger. The S2 trigger efficiency is measured by the fraction of S1-triggered events for which the S2 also triggered. The dependency of the trigger efficiency on the depth of the interaction in the TPC is illustrated in Figure 4 for the top, middle, and bottom of the TPC. At the analysis threshold of 200 PE, the trigger accepts over 98% of valid S2s in the center of the TPC.

A time window of $1~{}\mathrm{ms}$ is stored around each triggering S1 or S2 signal found above the trigger threshold, which was chosen to exceed the maximum drift time (670 $\upmu\mathrm{s}$) by several hundred $\upmu\mathrm{s}$ in order to always record some time before and after each event. Overlapping event windows are combined up to a maximum event size of 10 ms, at which point they are artificially truncated. This truncation causes a short amount of dead time, but since it only takes place for events that are over 15 drift lengths long, it is irrelevant for data acquired under normal conditions. The operating principle of the software trigger is presented in Figure 5.

The output of the event builder is a series of start and end times corresponding to event ranges. These are sent to a file-building system, which queries all pulses corresponding to the event time ranges, reformats them to the storage format, and writes out each event. This system consists of $\sim\!\!20$ worker processes on each of the three PCs that also host the MongoDB buffer databases, controlled via two RabbitMQ task queues [25]. The event builder puts event time ranges on a first queue, from which the workers read. The workers put the finished event objects in a second queue. Finally, a single process reads the event objects, then streams them to file in the order in which the events were acquired. At the XENON1T background event rate of $\sim\!\!5~{}\mathrm{Hz}$, the output data stream is approximately 30 GB/hour. During calibration, output rates up to a 250 GB/hour are possible for data collected without the high energy veto active.

4 Operation and Control

Besides the main processes reading and triggering data, various support systems are used to combine these processes into a working, controllable system and to keep track of the data collected. This section describes the runs database and web interface, shown as yellow boxes in Figure 1.

4.1 Run Database

The communication backend of the DAQ, and also of the data processing system not described here (see [10]), is another MongoDB database called the runs database. This is a three-member replica set with the primary at LNGS and two secondaries located at the University of Chicago and the University of Stockholm, which are the main XENON1T computing sites. Locating the replicas at the remote computing sites allows for local read access from computer nodes and seamless operation of many components of the processing and analysis systems independent of the LNGS network. Data recorded by the DAQ is organized into data sets called runs, where each run corresponds to data acquired for a given consecutive range of time (usually one hour) under identical settings. The runs database holds a record of all DAQ, trigger, and data processor settings, important operational conditions, operator comments, and an index of all locations where any type of data is stored for each run of the DAQ. Metadata for muon veto runs are also stored in this database in the same way. This index is used within the DAQ system for subsystem communication. The digitizer readout code, kodiaq, makes an entry at run start that it wrote untriggered data to the MongoDB buffer. The event builder registers this and begins triggering the run, while also making an entry that it began writing a data file to the local storage array at LNGS. The transfer system registers this and transfers the written raw data out to the storage and processing sites, and so on. The computing system uses the runs database as an index to track the location of all data associated with a particular run. For example, one run may have raw data written to tape archive, one to two additional copies of raw data in cloud storage, and one or more copies of processed data at analysis sites, all of which is tracked in the run document. Access to this database is also embedded into the XENON1T analysis software and analysts query the run database to select data with certain properties, for example data acquired with a specific calibration source or acquired with certain operating conditions.

4.2 User Interfaces

The DAQ system is accessed by users via a web interface with two entry points: the graphical interface, which is written in HTML and JavaScript for human interaction, and the API, which is for programmatic access. The backend is powered by the Django framework [26]. The human user interface allows configuration of the DAQ by experts into pre-defined readout modes that define the configuration for all parts of the system for calibration, dark matter search, and other data-taking. Both the TPC and muon veto subdetectors are controlled over this interface either independently or in synchronous operation. Operators can monitor the data rates, block transfer rates, and memory and CPU usage of each readout node. The software trigger also provides diagnostic information including its status, database buffer storage capacity, current run, and current event rate. A monitoring app allows inspection of the lone and coincident self-trigger rate per channel for any run and a live oscilloscope displays the most recent pulses recorded in any channel at 10 ns sampling rate. Both of these have proven instrumental for immediate remote diagnosis of detector anomalies.

A graphical interface to the runs database allows operators to apply single-word tags to mark exceptional runs (e.g., "test" for test runs not intended to be used in the dark matter exposure), add extended explanatory comments associated with single runs, view the processing status and data locations of all data types associated with a run, and browse and search runs by their properties. The interface also provides general logging functionality, user control, and an organizational platform for operation shifts. Figure 6 shows a screen shot from the control page of the web interface.

The API is implemented with the Django tastypie framework [27] and provides programmatic access to the runs database and DAQ backend. This API is world-accessible and is used by the computing and data transfer system to read and record processing statuses and storage locations for individual runs. The slow control system uses this API to query the DAQ status and monitor the system. Key parameters, including the data transfer rate, are also stored in the slow control databases. Disruptions within the DAQ are also propagated to the slow control system over the API, and alarms are raised in the case of DAQ errors, loss of connectivity to the API, or if the data rate has been zero for an extended time. The slow control alarm system then alerts operators and experts via email and telephone [14].

5 Conclusion

The XENON1T DAQ system takes advantage of the speed and data-structure of NoSQL databases to implement a flexible software trigger. At low data rates such as in dark matter search mode, this allows the trigger decision to be deferred, possibly indefinitely, or performed remotely. The flexible software trigger is made possible by the use of a custom digitizer firmware to stream all data from the detector to software without any reduction. Every photon measured by the PMTs is inserted as a document in the MongoDB buffer database and used to identify events.

Additionally, the use of a modern web application to control the experiment allows a greater degree of operational flexibility and has allowed operators to forego a conventional control center. Touch-screen tablets allow full control of the system and inform operators of disruption, day or night. This system is the precursor to another iteration, currently being designed and commissioned at the XENONnT experiment, which will further expand upon the strengths of the XENON1T DAQ by forgoing the trigger stage completely in favor of fast, online data processing [28].