Evaluation of Seismic Artificial Intelligence with Uncertainty

TL;DR

This paper introduces a comprehensive evaluation framework for seismic deep learning models that accounts for performance uncertainty and learning efficiency, aiding practitioners in model selection and performance assessment.

Contribution

It presents a novel evaluation framework that jointly considers uncertainty and efficiency, specifically tailored for seismic deep learning models, with a detailed data splitting and training design.

Findings

Framework effectively evaluates model performance with uncertainty.

Application to PhaseNet demonstrates robustness against misleading claims.

Guides practitioners in model choice and setting realistic expectations.

Abstract

Artificial intelligence has transformed the seismic community with deep learning models (DLMs) that are trained to complete specific tasks within workflows. However, there is still lack of robust evaluation frameworks for evaluating and comparing DLMs. We address this gap by designing an evaluation framework that jointly incorporates two crucial aspects: performance uncertainty and learning efficiency. To target these aspects, we meticulously construct the training, validation, and test splits using a clustering method tailored to seismic data and enact an expansive training design to segregate performance uncertainty arising from stochastic training processes and random data sampling. The framework's ability to guard against misleading declarations of model superiority is demonstrated through evaluation of PhaseNet [1], a popular seismic phase picking DLM, under 3 training approaches. Our framework helps practitioners choose the best model for their problem and set performance expectations by explicitly analyzing model performance with uncertainty at varying budgets of training data.