Pervasive Label Errors in Seismological Machine Learning Datasets

TL;DR

This paper identifies and analyzes pervasive label errors in seismological machine learning datasets, demonstrating how these inaccuracies can hinder model performance and proposing methods to improve data quality.

Contribution

It systematically evaluates common label errors in large seismological datasets using deep learning models, highlighting their impact and suggesting ways to enhance data reliability for better model training.

Findings

Average error rate of 3.9% across datasets

Four main types of label errors identified

Faulty data likely degrade model performance

Abstract

The recent boom in artificial intelligence and machine learning has been powered by large datasets with accurate labels, combined with algorithmic advances and efficient computing. The quality of data can be a major factor in determining model performance. Here, we detail observations of commonly occurring errors in popular seismological machine learning datasets. We used an ensemble of available deep learning models PhaseNet and EQTransformer to evaluate the dataset labels and found four types of errors ranked from most prevalent to least prevalent: (1) unlabeled earthquakes; (2) noise samples that contain earthquakes; (3) inaccurately labeled arrival times, and (4) absent earthquake signals. We checked a total of 8.6 million examples from the following datasets: Iquique, ETHZ, PNW, TXED, STEAD, INSTANCE, AQ2009, and CEED. The average error rate across all datasets is 3.9 %, ranging from nearly zero to 8 % for individual datasets. These faulty data and labels are likely to degrade model training and performance. By flagging these errors, we aim to increase the quality of the data used to train machine learning models, especially for the measurement of arrival times, and thereby to improve the reliability of the models. We present a companion list of examples that contain problems, aiming to integrate them into training routines so that only the reliable data is used for training.