How accurate are transient spectral classification tools? -- A study using 4,646 SEDMachine spectra

TL;DR

This study evaluates the accuracy of spectral classification tools for transients using a large dataset of 4,646 spectra, revealing their strengths and limitations to improve future classification efforts.

Contribution

It provides a comprehensive assessment of SNID, NGSF, and DASH accuracy on a large, homogeneous dataset, highlighting the need for supplementary human inspection.

Findings

NGSF achieves 87.6% overall accuracy for SNe Ia and Non-Ia types.

SNID accurately classifies SNe Ia with rlap > 15, purity > 98%.

All tools require human verification for uncertain classifications.

Abstract

Accurate classification of transients obtained from spectroscopic data are important to understand their nature and discover new classes of astronomical objects. For supernovae (SNe), SNID, NGSF (a Python version of SuperFit), and DASH are widely used in the community. Each tool provides its own metric to help determine classification, such as rlap of SNID, chi2/dof of NGSF, and Probability of DASH. However, we do not know how accurate these tools are, and they have not been tested with a large homogeneous dataset. Thus, in this work, we study the accuracy of these spectral classification tools using 4,646 SEDMachine spectra, which have accurate classifications obtained from the Zwicky Transient Facility Bright Transient Survey (BTS). Comparing our classifications with those from BTS, we have tested the classification accuracy in various ways. We find that NGSF has the best performance (overall Accuracy 87.6% when samples are split into SNe Ia and Non-Ia types), while SNID and DASH have similar performance with overall Accuracy of 79.3% and 76.2%, respectively. Specifically for SNe Ia, SNID can accurately classify them when rlap > 15 without contamination from other types, such as Ibc, II, SLSN, and other objects that are not SNe (Purity > 98%). For other types, determining their classification is often uncertain. We conclude that it is difficult to obtain an accurate classification from these tools alone. This results in additional human visual inspection effort being required in order to confirm the classification. To reduce this human visual inspection and to support the classification process for future large-scale surveys, this work provides supporting information, such as the accuracy of each tool as a function of its metric.