Transient processing and analysis using $\texttt{AMPEL}$: Alert Management, Photometry and Evaluation of Lightcurves

TL;DR

AMPEL is a flexible, real-time analysis framework for astrophysical transients that integrates alert management, photometry, and lightcurve evaluation, supporting high-throughput surveys and multi-messenger astronomy.

Contribution

It introduces a novel, adaptable system for transient analysis that combines alert brokerage with user-contributed code and provenance tracking, enabling efficient real-time and retrospective studies.

Findings

Successfully combined IceCube neutrino data with ZTF alerts in real time.

Reprocessed four months of ZTF alerts, confirming detection completeness.

Identified three key channels for transient collection and follow-up.

Abstract

Both multi-messenger astronomy and new high-throughput wide-field surveys require flexible tools for the selection and analysis of astrophysical transients. We here introduce the Alert Management, Photometry and Evaluation of Lightcurves (AMPEL) system, an analysis framework designed for high-throughput surveys and suited for streamed data. AMPEL combines the functionality of an alert broker with a generic framework capable of hosting user-contributed code, that encourages provenance and keeps track of the varying information states that a transient displays. The latter concept includes information gathered over time and data policies such as access or calibration levels. We describe a novel ongoing real-time multi-messenger analysis using AMPEL to combine IceCube neutrino data with the alert streams of the Zwicky Transient Facility (ZTF). We also reprocess the first four months of ZTF public alerts, and compare the yields of more than 200 different transient selection functions to quantify efficiencies for selecting Type Ia supernovae that were reported to the Transient Name Server (TNS). We highlight three channels suitable for (1) the collection of a complete sample of extragalactic transients, (2) immediate follow-up of nearby transients and (3) follow-up campaigns targeting young, extragalactic transients. We confirm ZTF completeness in that all TNS supernovae positioned on active CCD regions were detected. AMPEL can assist in filtering transients in real time, running alert reaction simulations, the reprocessing of full datasets as well as in the final scientific analysis of transient data. This text introduces how users can design their own channels for inclusion in the AMPEL live instance that parses the ZTF stream and the real-time submission of high quality extragalactic supernova candidates to the TNS.

Tables

Table 1. Table 1: AMPEL terminology

Term	AMPEL interpretation
Transient	Object with a unique ID provided from a data source and accepted into AMPEL by at least one channel.
Datapoint	A single measurement with a specific calibration, processing level etc.
Compound	A collection of datapoints (from one or more instruments).
State	A view of a transient object available at some point for some observer. Connects a compound with one (or more) transients.
Tier	AMPEL is internally divided into four tiers, where each performs different kinds of operations and is controled by a separate scheduler.
Channel	Configuration of requested behaviour at all AMPEL tiers supplied by a user (for one science goal). Typically consists of a list of requested units together with their run parameters.
Archive	All alert data, also those rejected during live processing, are stored in an archive for reprocessing.
ScienceRecord	Records the result of a science computation made based on data available in specific state.
TransientView	All information available regarding a specific transient. This can include multiple states, and any ScienceRecords associated with these.
Unit	Typically implemented as python modules, a unit allows user contributed code to be directly called during data processing. Units at different tiers receive different input and are expected to produce different kinds of output.
Journal	A time-ordered log included in each transient.
Purge	The transfer of a no longer active transient from the live database to external storage. This includes all connected datapoints, states, compounds and ScienceRecords.
Live instance	A version of AMPEL processing data in real-time. This includes a number of active channels.
Reprocessing	Parsing archived alerts as they would have been received in real-time, using a set of channels defined as for a live instance.

Table 2. Table 2: Dominant channel selection variables and potential settings.

Channel property	Options
RealBogus	Nominal: Require ML score above $0.3$ or Strong: above $0.5$
Detections	More than $[2,4,6,8]$ (any filter)
Alert History	New: Not older than $5$ days, Multi-night: $4$ to $15$ days, Persistent: Older than $8$ days.
Image Quality	All: No requirements, Good: Limited cuts on e.g. FWHM and bad pixels, Excellent: Strong cuts on e.g. FWHM and bad pixels.
Gaia DR2	Nominal: Reject likely stars from Gaia DR2, Moderate: only search in small aperture or Disabled.
Star-Galaxy separation	Using PS1 star-galaxy separation (Tachibana & Miller 2018) to reject potential stars (Hard), likely stars (Nominal) or no rejection (Disabled).
Match confusion	Nominal: Allow candidates close to nearby (confused) sources, or Disabled: reject anything close to stars even if other sources exist.

Table 3. Table 3: AMPEL sample channel parameter settings and rerun statistics. Columns 2 2 2 to 6 6 6 show settings used for parameters in Table  2 , columns 7 7 7 to 10 10 10 statistics including all targets and columns 11 11 11 to 14 14 14 repeating these when excluding AGN associated candidates. The phase estimates describe the fraction of the matched SNe Ia with a good peak phase estimate that were accepted by the channel either prior to lightcurve peak or prior to − 10 10 -10 days with respect to peak light.

Channel	RealBogus	Detections	History	Image	Gaia	SNe Ia	Detections	Phase()	Phase()	SNe Ia	Detections	Phase()	Phase()
						(all)	(all)	(all)	(all)	(no AGN)	(no AGN)	(no AGN)	(no AGN)
11	0.5	4	multinight	good	nominal	125	1857	0.956	0.067	120	1514	0.953	0.047
26	0.3	4	new	excellent	nominal	36	293	1.000	0.296	35	257	1.000	0.308
34	0.5	2	multinight	excellent	nominal	128	2479	0.944	0.079	123	1964	0.941	0.059
18	0.5	6	new	all	nominal	2	34	0.000	0.000	2	28	0.000	0.000
77	0.3	4	persistent	all	moderate	131	10672	0.832	0.000	126	9705	0.824	0.000
51	0.3	4	new	good	nominal	39	357	1.000	0.321	37	311	1.000	0.308
57	0.3	4	persistent	good	nominal	130	3078	0.830	0.000	125	2351	0.822	0.000
64	0.3	8	multinight	good	nominal	76	580	0.554	0.000	74	524	0.556	0.000
1	0.3	2	new	good	nominal	110	2968	0.987	0.557	106	2562	0.987	0.533
28	0.5	2	new	excellent	nominal	95	1833	0.985	0.485	92	1547	0.985	0.462
4	0.5	2	new	good	nominal	103	2286	0.986	0.514	99	1909	0.985	0.485
10	0.5	2	multinight	good	nominal
59	0.3	4	persistent	good	moderate
10+59						134	11112	0.926	0.084	129	9973	0.923	0.055

Table 4. Table 4: AMPEL sample channel parameter settings and rerun statistics for cases when only transients close to z < 0.1 𝑧 0.1 z<0.1 host galaxies are included. Columns 2 2 2 to 6 6 6 show settings used for parameters in Table  2 , columns 7 7 7 to 10 10 10 statistics including all targets and columns 11 11 11 to 14 14 14 repeating these when excluding AGN associated candidates. The phase estimates describe the fraction of the matched SNe Ia with a good peak phase estimate that were accepted by the channel either prior to lightcurve peak or to − 10 10 -10 days w.r.t. peak.

Channel	RealBogus	Detections	History	Image	Gaia	SNe Ia	Detections	Phase()	Phase()	SNe Ia	Detections	Phase()	Phase()
						(all)	(all)	(all)	(all)	(no AGN)	(no AGN)	(no AGN)	(no AGN)
85	0.3	4	persistent	good	nominal	21	261	0.750	0.000	19	143	0.714	0.000
34	0.5	2	multinight	excellent	nominal	20	231	0.867	0.067	18	136	0.846	0.077
16	0.5	2	new	all	nominal	17	160	0.923	0.385	15	101	0.909	0.273
29	0.5	4	new	excellent	nominal	4	9	1.000	0.000	3	6	1.000	0.000
84	0.3	8	multinight	all	moderate	11	32	0.333	0.000	10	24	0.375	0.000
28	0.5	2	new	excellent	nominal	15	117	0.917	0.333	14	79	0.909	0.273

Full text

*Aims. *We here introduce the Alert Management, Photometry and Evaluation of Lightcurves (AMPEL) system, an analysis framework designed for high-throughput surveys and suited for streamed data. AMPEL combines the functionality of an alert broker with a generic framework capable of hosting user-contributed code, that encourages provenance and keeps track of the varying information states that a transient displays. The latter concept includes information gathered over time and data policies such as access or calibration levels.

*Methods. *We describe a novel ongoing real-time multi-messenger analysis using AMPEL to combine IceCube neutrino data with the alert streams of the Zwicky Transient Facility (ZTF). We also reprocess the first four months of ZTF public alerts, and compare the yields of more than 200 different transient selection functions to quantify efficiencies for selecting Type Ia supernovae that were reported to the Transient Name Server (TNS).

*Results. *We highlight three channels suitable for (1) the collection of a complete sample of extragalactic transients, (2) immediate follow-up of nearby transients and (3) follow-up campaigns targeting young, extragalactic transients. We confirm ZTF completeness in that all TNS supernovae positioned on active CCD regions were detected.

*Conclusions. *AMPEL can assist in filtering transients in real time, running alert reaction simulations, the reprocessing of full datasets as well as in the final scientific analysis of transient data. This is made possible by a novel way to capture transient information through sequences of evolving states, and interfaces that allow new code to be natively applied to a full stream of alerts. This text also introduces how users can design their own channels for inclusion in the AMPEL live instance that parses the ZTF stream and the real-time submission of high quality extragalactic supernova candidates to the TNS.

1 Introduction

Transient astronomy has traditionally used optical telescopes to detect variable objects, both within and beyond our Galaxy, with a peak sensitivity for events that vary on weekly or monthly timescales. This field has now entered a new phase in which multi-messenger astronomy allows for near real-time detections of transients through correlations between observations of different messengers. The initial report of GW170817 from LIGO/VIRGO, and the subsequent search and detection of an X-ray/optical counterpart, provides a first, inspiring example of this (Abbott et al. 2017). Shortly after, the observation of a flaring blazar coincident with a high-energy neutrino detected by IceCube illustrated again the scientific potential of time domain multi-messenger astronomy (IceCube Collaboration et al. 2018). Optical surveys now observe the full sky daily, to a depth which encompasses both distant, bright objects and nearby, faint ones. We can thus simultaneously find rare objects, obtain an accounting of the variable Universe, and probe fundamental physics at scales beyond the reach of terrestrial accelerators. Exploiting these opportunities is currently constrained as much by software and method development as by available instruments (Allen et al. 2018).

The plans for the Large Synoptic Survey Telescope (LSST) provide a sample scale for high-rate transient discovery. LSST is expected to scan large regions of the sky to great depth, with sufficient cadence for more than $10^{6}$ astrophysical transients to be discovered each night. Each such detection will be immediately streamed to the community as an alert. The challenge of distributing this information for real-time follow-up observations is to be solved through a set of brokers, which will receive the full data flow and allow end-users to select the small subset that merits further study (Jurić et al. 2017). Development first started on the Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES), which provides a system for real-time characterization and annotation of alerts before they are relayed further downstream (Saha et al. 2014). Other current brokers include MARS 111https://mars.lco.global/ and LASAIR (Smith et al. 2019). Earlier systems for transient information distribution include the Central Bureau for Astronomical Telegrams (CBAT), the Gamma-ray Coordinates Network and the Astronomer’s Telegram. The Catalina Real-Time Transient Survey was deisgned to make transient detections public within minutes of observation (Drake et al. 2009; Mahabal et al. 2011). More recent developments include the Astrophysical Multimessenger Observatory Network (AMON, Smith et al. 2013), which provides a framework for real-time correlation of transient data streams from different high-energy observatories, and the Transient Name Server (TNS), which maintains the current IAU repository for potential and confirmed extragalactic transients222https://wis-tns.weizmann.ac.il/.

While LSST will come online only in 2022, the Zwicky Transient Facility (ZTF) has been operating since March 2018 (Graham 2019). ZTF employs a wide-field camera mounted on the Palomar P48 telescope, and is capable of scanning more than $3750$ square degrees to a depth of $20.5$ mag each hour (Bellm et al. 2019). This makes ZTF a wider, shallower precursor to LSST, with a depth more suited to spectroscopic follow-up observations. ZTF observations are immediately transferred to the Infrared Processing & Analysis Center (IPAC) data center for processing and image subtraction (Masci et al. 2019). Any significant point source-like residual flux in the subtracted image triggers the creation of an alert. Alerts are serialized and distributed through a Kafka333https://kafka.apache.org server hosted at the DiRAC centre at University of Washington (Patterson et al. 2019). Each alert contains primary properties like position and brightness, but also ancillary detection information and higher-level derived values such as the RealBogus score which aims to distinguish real detections from image artifacts (Mahabal et al. 2019). Full details on the reduction pipeline and alert content can be found in Masci et al. (2019), while an overview of the information distribution can be found in the top row of Fig. 3. ZTF will conduct two public surveys as part of the US NSF Mid-Scale Innovations Program (MSIP). One of these, the Northern Sky Survey, performs a three-day cadence survey in two bands of the visible Northern Sky.

We here present AMPEL (Alert Management, Photometry and Evaluation of Lightcurves) as a tool to accept, process and react to streams of transient data. AMPEL contains a broker as the first of four pipeline levels, or ’tiers’, but complements this with a framework enabling analysis methods to be easily and consistently applied to large data volumes. The same set of input data can be repeatedly reprocessed with progressively refined analysis software, while the same algorithms can then also be applied to real-time, archived and simulated data samples. Analysis and reaction methods can be contributed through the implementation of simple python classes, ensuring that the vast majority of current community tools can be immediately put to use. AMPEL functions as a public broker for use with the public ZTF alert stream, meaning that community members can provide analysis units for inclusion in the real-time data processing. AMPEL also brokers alerts for the private ZTF partnership. Selected transients, together with derived properties, are pushed into the GROWTH Marshal (Kasliwal et al. 2019) for visual examination, discussion and the potential trigger of follow-up observations.

This paper is structured as follows: AMPEL requirements are first described in Sec. 2, after which the design concepts are presented in Sec. 3, some specific implementation choices detailed in Sec. 4 and instructions for using AMPEL are provided in Sec. 5. In Sec. 6 we present sample AMPEL uses: systematic reprocessing of archived alerts to investigate transient search completeness and efficiency, photometric typing and live multi-messenger matching between optical and neutrino data-streams. The discussion (Sec. 7) introduces the automatic AMPEL submission of high-quality extragalactic astronomical transients to the TNS, from which astronomers can immediately find potential supernovae or AGNs without having to do any broker configuration. The material presented here focuses on the design and concepts of AMPEL, and acts as a complement to the software design tools contained in the AMPEL sample repository444https://github.com/AmpelProject/Ampel-contrib-sample. We encourage the interested reader to look at this in parallel to this text. We describe the AMPEL system using terms where the interpretation might not match that used in other fields. This terminology will be introduced gradually in this text, but is summarized in Table 1 for reference.

2 Requirements

Guided by an overarching goal of analyzing data streams, we here lay out the design requirements that shaped the AMPEL development:

Provenance and reproducibility:

Data provenance encapsulates the philosophy that the origin and manipulation of a dataset should be easily traceable. As data volumes grow, and as astronomers increasingly seek to combine ever more diverse datasets, the concept of data provenance will be of central importance. In this era, individual scientists can be expected neither to master all details of a given workflow, nor to inspect all data by hand. As an alternative, these scientists must instead rely on documentation accompanying the data. While provenance is a minimal requirement for such analysis, a more ambitious goal is replayability. Replaying an archival transient survey offline would involve providing a virtual survey in which the entire analysis chain is simulated, from transient detection to the evaluation of triggered follow-up observations. In essence, this amounts to answering the question: If I had changed my search or analysis parameters, what candidates would have been selected? Because any given transient will only be observed once, replayability is as close to the standard scientific goal of reproducibility as astronomers can get.

Analysis flexibility:

The next decades will see an unprecedented range of complementary surveys looking for transients through gravitational waves, neutrinos and multiwavelength photons. These will feed a sprawling community of diverse science interests. We would like a transient software framework that is sufficiently flexible to give full freedom in analysis design, while still being compatible with existing tools and software.

Versions of data and software:

It is typical that the value of a measurement evolves over time, from a preliminary real-time result to final published data. This is driven both by changes in the quantitative interpretation of the observations, as well as a progressive increase in analysis complexity. The first dimension involves changes such as improved calibration, while the second incorporates, for example, more computationally expensive studies only run on subsets of the data. So far it has been hard to study the full impact of incremental changes in these two dimensions. To change this requires an end-to-end streaming analysis framework where any combinations of data and software can be conveniently explored. A related community challenge is to recognize, reference and motivate continued development of well-written software.

Alert rate:

Current optical transient surveys such as DES, ZTF, ASAS-SN and ATLAS, as well as future ones (LSST), do or will provide tens of thousands to millions of detections each night. With such scale, it is impossible for human inspection of all candidates, even assuming that artifacts could be perfectly removed555For optical surveys, a majority of these “detections” are actually artifacts induced through the subtraction of a reference image. Machine learning techniques, such as RealBogus for ZTF, are increasingly powerful at separating these from real astronomical transients. However, this separation can never be perfect and any transient program has to weigh how aggressively to make use of these classifications.. A simplistic solution to this problem is to only select a very small subset from the full stream, for example a handful of the brightest objects, for which additional human inspection is feasible. A more complete approach would be based on retaining much larger sets of targets throughout the analysis, from which subsets are complemented with varying levels of follow-up information. As the initial subset selection will by necessity be done in an automated streaming context, the accompanying analysis framework must be able to trace and model these real-time decisions.

3 AMPEL in a nutshell

AMPEL is a framework for analyzing and reacting to streamed information, with a focus on astronomical transients. Fulfilling the above design goals requires a flexible framework built using a set of general concepts. These will be introduced in this section, accompanied by examples based on optical data from ZTF. The “life” of a transient in AMPEL is in parallel outlined in Figs. 1 and 2. These further illustrate many of the concepts introduced in this section. Fig. 1 shows AMPEL used as a straightforward alert broker, while Fig. 2 includes many of the additional features that make AMPEL into a full analysis framework.

The core object in AMPEL is a transient, a single object identified by a creation date and typically a region of origin in the sky. Each transient is linked to a set of datapoints that represent individual measurements666Note that this is a many-to-many connection; multiple transients can be connected to the same datapoint due to e.g. positional uncertainty. Datapoints can also originate from different sources.. Datapoints can be added, updated or marked as bad. Datapoints are never removed. Each datapoint can be associated with tags indicating e.g. any masking or proprietary restrictions. Transients and datapoints are connected by states, where a state references a compound of datapoints. A state represents a view of a transient available at some time and for some observer. For an optical photometric survey, a compound can be directly interpreted as a set of flux measurements or a lightcurve.

*Example: A ZTF alert corresponds to a potential transient. Datapoints here are simply the photometric magnitudes reported by ZTF, which in most cases consists of a recent detection and a history of previous detections or non-detections at this position. When first inserted, a transient has a single state with a compound consisting of the datapoints in the initial alert. Should a new alert be received with the same ZTF ID, the new datapoints contained in this alert are added to the collection and a new state is created containing both previous and new data. Should the first datapoint be public but the second datapoint be private, only users with proper access will see the updated state. *

Using AMPEL means creating a channel, corresponding to a specific science goal, which prescribes behavior at four different stages, or tiers. What tasks should be performed at what tier can be determined by answers to the questions: “Tier 0: What are the minimal requirements for an alert to be interesting?”, “Tier 1: Can datapoints be changed by events external to the stream?”, “Tier 2: What calculations should be done on each of the candidates states?”, “Tier 3: What operations should be done at timed intervals or on populations of transients?”777Timed intervals include very high frequencies or effectively real-time response channels.

• •

Tier 0 (T0) filters the full alert stream to only include potentially interesting candidates. This tier thus works as a data broker: objects that merit further study are selected from the incoming alert stream. However, unlike most brokers, accepted transients are inserted into a database (DB) of active transients rather than immediately being sent downstream. All alerts, also those rejected, are stored in an external archive DB. Users can either provide their own algorithm for filtering, or configure one of the filter classes already available according to their needs.

*Example T0: The simple AMPEL channel “BrightAndStable” looks for transients with at least three well behaved detections (few bad pixels and reasonable subtraction FWHM) and not coincident with a Gaia DR2 star-like source. This is implemented through a python class SampleFilter that operates on an alert and returns either a list of requests for follow-up (T2) analysis, if selection criteria are fulfilled, or False if they are not. AMPEL will test every ZTF alert using this class, and all alerts that pass the cut are added to a DB containing all active transients. The transient is there associated with the channel “BrightAndStable”. *

• •

Tier $1$ (T1) is largely autonomous and exists in parallel to the other tiers. T1 carries out duties of assigning datapoints and T2 run requests to transient states. Example activities include completing transient states with datapoints that were present in new alerts but where these were not individually accepted by the channel filter (e.g., in the case of lower significance detections at late phases), as well as querying an external archive for updated calibration or adding photometry from additional sources. A T1 unit could also parse previous alerts at or close to the transient position for old data to include with the new detection.

• •

Tier $2$ (T2) derives or retrieves additional transient information, and is always connected to a state and stored as a ScienceRecord. T2 units either work with the empty state, relevant for e.g. catalog matching that only depends on the position, or they depend on the datapoints of a state to calculate new, derived transient properties. In the latter case, the T2 task will be called again as soon as a new state is created. This could be due both to new observations or, for example, updated calibration of old datapoints. Possible T2 units include lightcurve fitting, photometric redshift estimation, machine learning classification, and catalog matching.

*Example T2: For an optical transient, a state corresponds to a lightcurve and each photometric observation is represented by a datapoint. A new observation of the transient would extend the lightcurve and thus create a new state.“BrightAndStable” requests a third order polynomial fit for each state using the T2PolyFit class. The outcome, in this case polynomial coefficients, are saved to the database. *

• •

Tier $3$ (T3), the final AMPEL level, consists of schedulable actions. While T2s are initiated by events (the addition of new states), T3 units are executed at pre-determined times. These can range from yearly data dumps, to daily updates or to effectively real-time execution every few seconds. T3 processes access data through the TransientView, which concatenates all information regarding a transient. This includes both states and ScienceRecords that are accessible by the channel. T3s iterate through all transients of a channel which were updated since a previous timestamp (either the last time the T3 was run or a specified time-range). This allows for an evaluation of multiple ScienceRecords and comparisons between different objects (such as any kind of population analysis). One typical case is the ranking of candidates which would be interesting to observe on a given night. T3 units include options to push and pull information to and from for example the TNS, web-servers and collaboration communication tools such as Slack888https://slack.com.

*Example T3: The science goal of “BrightAndStable” is to observe transients with a steady rise. At the T3 stage the channel therefore loops through the TransientViews, and examines all T2PolyFit science records for fit parameters which indicate a lasting linear rise. Any transients fulfilling the final criteria trigger an immediate notification sent to the user. This test is scheduled to be performed at 13:15 UTC each day. *

4 Implementation

We here expand on a selection of implementational aspects. An overview of the live instance processing of the ZTF alert stream can be found in Fig. 3.

Modularity and Units

Modularity is achieved through a system of units. These are python modules that can be incorporated with AMPEL and directly be applied to a stream of data. Units are inherited from abstract classes that regulate the input and output data format, but have great freedom in implementing what is done with the input data. The tiers of AMPEL are designed such that each requires a specific kind of information: At Tier [math] the input is the raw alert content, at Tier $2$ a transient state, and at Tier $3$ a transient view. The system of base classes allows AMPEL to provide each unit with the required data. In a similar system, each unit is expected to provide output data (results) in a specific format to make sure this is stored appropriately: At Tier [math] the expected output is a list of Tier $2$ units to run at each state for accepted transients (None for rejected transients). At Tier $2$ output is a science record (dictionary) which in the DB is automatically linked to the state from which it was derived. The T3 output is not state-bound, but is rather added to the transient journal, a time-ordered history accompanying each transient. Modules at all tiers can make direct use of well developed libraries such as numpy (Oliphant 2006–), scipy (Jones et al. 2001–), and astropy (Astropy Collaboration et al. 2013; Price-Whelan et al. 2018). Developers can choose to make their contributed software available to other users, and gain recognition for functional code, or keep them private. The modularity means that users can independently vary the source of alerts, calibration version, selection criteria and analysis software.

Schemas and AMPEL shapers

Contributed units will be limited as long as they have to be tuned for a specific kind of input, e.g., ZTF photometry. Eventually, we hope that more general code can be written through the development of richer schemas for astronomical information based on which units can be developed and immediately applied to different source streams. The International Virtual Observatory Alliance (IVOA) initiated the development of the VOEvent standard with this purpose999http://www.ivoa.net/documents/VOEvent/20110711/REC-VOEvent-2.0.pdf. Core information of each event is to be mapped to a set of specific tags (such as Who, What, Where, When), stored in an XML document. VOEvents form a starting point for this development (see e.g. Williams et al. 2009), but more work is needed before a general T2 unit can be expected to immediately work on data from all sources. As an intermediate solution, AMPEL employs shapers that can translate source-specific parameters to a generalized data format that all units can rely on. While the internal AMPEL structure is designed for performance and flexibility, it is easy to construct T3 units that export transient information according to, for example, VOEvent or GCN specifications.

The archive

Full replayability requires that all alerts are available at later times. While most surveys are expected to provide this, we keep local copies of all alerts until other forms of access are guaranteed.

Catalogs, Watch-lists and ToO triggers

Understanding astronomical transients frequently requires matches to known source catalogs. AMPEL currently provides two resources to this end. A set of large, pre-packaged catalogs can be accessed using catsHTM, including the Gaia DR2 release (Soumagnac & Ofek 2018). As a complement, users can upload their own catalogs using extcats101010https://github.com/MatteoGiomi/extcats for either transient filtering or to annotate transients with additional information. extcats is also used to create watch-lists and ToO channels. Watchlists are implemented as a T0 filter that matches the transient stream with a contributed extcat catalog. A ToO channel has a similar functionality, but employs a dynamic extcat target list where a ToO trigger immediately adds one or more entries to the matchlist. The stream can in this case initially be replayed from some previous time (a delayed T0), which allows preexisting transients to be consistently detected.

The live database

The live transient DB is built using the NoSQL MongoDB111111https://docs.mongodb.com/manual/ engine. The flexibility of not having an enforced schema allows AMPEL to integrate varying alert content and give full freedom to algorithms to provide output of any shape. The live AMPEL instance is a closed system that users cannot directly interact with, and contributed units do not directly interact with the DB. Instead, the AMPEL core system manages interactions through the alert, state and transient view objects introduced above121212Eventually, daily snapshot copies of the DB will be made available for users to interactively examine the latest transient information without being limited with what was reconfigured to be exported.. Each channel also specifies conditions for when a transient is no longer considered “live”. At this point it is purged: extracted from the live DB together with all states, computations and logs, and then inserted into a channel specific offline DB which is provided to the channel owner.

Horizontal scaling

AMPEL is designed to be fully parallelizable. The DB, the alert processors and tier controllers all scale horizontally such that additional workers can be added at any stage to compensate for changes to the workload. Alerts can be processed in any order, i.e. not necessarily in time-order.

AMPEL instances and containers

An AMPEL instance is created through combining tagged versions of core and contributed units into a Docker (Merkel 2014) image, which is then converted to the Singularity (Kurtzer et al. 2017) format for execution by an unprivileged user. The final product is a unique “container” that is immutable and encapsulates the AMPEL software, contributed units and their dependencies. These can be reused and referenced for later work, even if the host environment changes significantly. The containers are coordinated with a simple orchestration tool131313https://github.com/AmpelProject/singularity-stack that exposes an interface similar to Docker’s “swarm mode.” Previously-deployed AMPEL versions are stored, and can be run off-line on any sequence of archived or simulated alerts. Several instances of AMPEL might be active simultaneously, with each processing either a fraction of a full live-stream, or some set of archived or simulated alerts. Each works with a distinct database. The current ZTF alert flow can easily be parsed by a single instance, called the live instance. A full AMPEL analysis combines this active parsing and reacting to the live streams with subsequent or parallel runs in which the effects of the channel parameters can be systematically explored.

Logs and provenance

AMPEL contains extensive, built-in logging functions. All AMPEL units are provided a logger, and we recommend this to be consistently used. Log entries are automatically tagged with the appropriate channel and transient ID, and are then inserted into the DB. These tools, together with the DB content, alert archive and AMPEL container, make provenance straightforward. The IVOA has initiated the development of a Provenance Data Model (DM) for astronomy, following the definitions proposed by the W3C (Sanguillon et al. 2017)141414http://www.ivoa.net/documents/ProvenanceDM/20181015/PR-ProvenanceDM-1.0-20181015.pdf. Scientific information is here described as flowing between agents, entities and activities. These are related through causal relations. The AMPEL internal components can be directly mapped to the categories of the IVOA Provenance DM: Transients, datapoints, states and ScienceRecords are entities, Tier units are activities and users, AMPEL maintainers, software developers and alert stream producers are agents. A streaming analysis carried out in AMPEL will thus automatically fulfill the IVOA provenance requirements.

Hardware requirements

The current live instance installed at the DESY computer center in Berlin-Zeuthen consists of two machines, “Burst” and “Transit”. Real time alert processing is done at Burst (32 cores, 96 GB memory, 1 TB SSD) while alert reception and archiving is done at Transit (20 cores, 48 GB memory, 1 TB SSD + medium-time storage). This system has been designed for extragalactic programs based on the ZTF survey, with a few $\times 10^{5}$ alerts processed each night, of which between $0.1$ and $1\%$ are accepted. Reprocessing large alert volumes from the archive on Transit is done at a mean rate of $100$ alerts per second. As the ZTF live alert production rate is lower than this, and Burst is a more powerful machine, this setup is never running at full capacity. It would be straightforward to distribute processing of T2 and T3 tasks among multiple machines, but as the expected practical limitation is access to a common database, this is of limited use until extremely demanding units are requested.

5 Using AMPEL

5.1 Creating a channel for the ZTF alert stream

The process for creating AMPEL units and channels is fully described in the Ampel-contrib-sample repository151515 https://github.com/AmpelProject/Ampel-contrib-sample , which also contains a set of sample channel configurations. The steps to implementing a channel can be summarized as follows:

Fork the sample repository and rename it Ampel-contrib-groupID where groupID is a string identifying the contributing science team. 2. 2.

Create units through populating the t0/t2/t3 sub-directories with python modules. Each is designed through inheriting from the appropriate base class. 3. 3.

Construct the repository channels by defining their parameters in two configuration files: channels.json which defines the channel name and regulates the T0, T1 and T2 tiers, and t3_jobs.json which determines the schedule for T3 tasks. These can be constructed to make use of AMPEL units present either in this repository, or from other public AMPEL repositories. 4. 4.

Notify AMPEL administrators. The last step will trigger channel testing and potential edits. After the channel is verified, it will be added in the list of AMPEL contribution units and included in the next image build. The same channel can also (or exclusively) be applied to archived ZTF alerts.

5.2 Using AMPEL for other streams

Nothing in the core AMPEL design is directly tied to the ZTF stream, or even optical data. The only source-specific software class is the Kafka client reading the alert stream, and the alert shapers which make sure key variables such as coordinates are stored in a uniform matter. Using a schema-free DB means that any stream content can be stored by AMPEL for further processing. A more complex question concerns how to design units usable with different stream sources. As an example, different optical surveys might use different conventions for how to encode which filter was used, and the reference system and uncertainty of reported magnitudes and powerful alert metrics, such as the RealBogus value of ZTF, are unique. Until common standards are developed, classes will have to be tuned directly to every new alert stream.

6 Initial AMPEL applications

6.1 Exploring the ZTF alert parameter space

It has been notoriously challenging to quantify transient detection efficiencies, search old surveys for new kinds of transients, and predict the likely yield from a planned follow-up campaign. We here demonstrate how AMPEL can assist with such tasks. For this case study we reprocess $4$ months of public ZTF alerts using a set of AMPEL filters spanning the parameter space of the main properties of ZTF alerts. The accepted samples of each channel are, in a second step, compared with confirmed Type Ia supernovae (SNe Ia) reported to the TNS during the same period. We can thus examine how different channel permutations differ in detection efficiency, and at what phase each SN Ia was “discovered”. The base comparison sample consists of $134$ normal SNe Ia. The creation of this sample is described in detail in Appendix A.

We processed the ZTF alert archive from June 2nd 2018 (start of the MSIP Northern Sky Survey) to October 1st using $90$ potential filter configurations based on the DecentFilter class. In total $28667252$ alerts were included. Each channel exists on a grid constructed by varying the properties described in Table 2. We also include $24$ OR combinations where the accept criteria of two filters are combined. We further consider two additional versions of each filter or filter-combination:

Transients in galaxies with known active SDSS or MILLIQUAS active nuclei (Flesch 2015; Pâris et al. 2017) are rejected, and 2. 2.

Transients are required to be associated with a galaxy for which there is a known NED or SDSS spectroscopic redshift $z<0.1$.

In total, this amounts to 342 combinations. All of these variants include some version of alert rejection based on coincidence with a star-like object in either PanSTARRS (using the algorithm of Tachibana & Miller 2018) or Gaia DR2 (Gaia Collaboration et al. 2018). We also tested channels not including any such rejection, which lead to transient counts around $10^{5}$ (an order of magnitude greater than with the star-rejection veto). Reprocessing the alert stream in this way took $5$ days even in a non-optimized configuration, demonstrating that AMPEL can process data at the expected LSST alert rate.

This study is neither complete nor unbiased: A large fraction of the SNe were classified by ZTF, and we know that the real number of SNe Ia observed is much larger than the classified subset. Nonetheless it serves both as a benchmark test for channel creation, as well as a starting point for more thorough analysis. An estimate of the total number of supernovae we expect to be hidden in the ZTF detections can be obtained through the simsurvey code (Feindt et al. 2019), in which known transient rates are combined with a realistic survey cadence and a set of detection thresholds161616https://github.com/ufeindt/simsurvey. The predicted number of Type Ia supernovae fulfilling the criteria of one or more of these channels over the same timespan as the comparison sample and with weather conditions matching the observed was found to be $1033$ (average over 10 simulations). Simsurvey also conveniently returns estimates for other supernova types and we find that an additional $276$ Type Ibc, $92$ Type IIn and $377$ Type IIP supernovae are likely to have been observed by ZTF under the same conditions. The total number of supernovae present in the alert sample is thus estimated to be $1778$.

The results for channel efficiencies, compared to the total number of accreted transients, can be found in Fig. 4. Though we observe the obvious trend that channels with larger coverage of the comparison sample also accept a larger total number of transients, there is also a variation in the total transient counts between configurations that find the same fraction of the comparison sample. Fig. 4 highlights a subset of the channels as particularly interesting. Selection statistics for these channels can be found in Table 6.1. For comparison objects with a well defined time of peak light, we also monitor the phase at which it was accepted into each channel. As an estimate for this we use the time of B-band peak light as determined by a SALT lightcurve fit, which is carried out for each candidate at the T2 tier (Betoule et al. 2014). This information can be used to study how well channels perform in finding early SNe Ia, which constitute a prime target for many supernova science studies. In figure 4 we therefore mark all channels where more than $25\%$ of all SNe Ia were accepted prior to $-10$ days relative to peak light (“Early detection”). Alternatively, SN Ia cosmology programs often look for a combination of completeness and discovery around lightcurve peak to facilitate spectroscopic classification. Channels not fulfilling the Early detection criteria but where more than $95\%$ of all SNe Ia were accepted prior to peak light are therefore marked as “Peak classification”. These two simple examples highlight how reprocessing alert streams (reruns) can be used to optimize transient programs, and to estimate yields useful for e.g follow-up proposals. We also find that a $4\%$ fraction of the comparison sample ($5$ out of $134$) were found in galaxies with documented AGNs, suggesting that programs which prioritize supernova completeness cannot reject nuclear transients with active hosts.