CRACKS: Crowdsourcing Resources for Analysis and Categorization of Key Subsurface faults

TL;DR

This paper introduces the CRACKS dataset, leveraging crowdsourcing to annotate and analyze subsurface faults in imaging data, aiming to facilitate fault detection with diverse annotator expertise and confidence levels.

Contribution

The paper presents a novel crowdsourced dataset for fault segmentation in subsurface images, including annotations from novices, practitioners, and an expert, with benchmarks for detection and segmentation tasks.

Findings

Crowdsourcing can generate valuable fault annotations from diverse skill levels.

Annotations include confidence levels, enabling analysis of annotation reliability.

Benchmark results show the potential for automated fault detection using crowdsourced data.

Abstract

Crowdsourcing annotations has created a paradigm shift in the availability of labeled data for machine learning. Availability of large datasets has accelerated progress in common knowledge applications involving visual and language data. However, specialized applications that require expert labels lag in data availability. One such application is fault segmentation in subsurface imaging. Detecting, tracking, and analyzing faults has broad societal implications in predicting fluid flows, earthquakes, and storing excess atmospheric CO$_2$. However, delineating faults with current practices is a labor-intensive activity that requires precise analysis of subsurface imaging data by geophysicists. In this paper, we propose the $\texttt{CRACKS}$ dataset to detect and segment faults in subsurface images by utilizing crowdsourced resources. We leverage Amazon Mechanical Turk to obtain fault delineations from sections of the Netherlands North Sea subsurface images from (i) $26$ novices who have no exposure to subsurface data and were shown a video describing and labeling faults, (ii) $8$ practitioners who have previously interacted and worked on subsurface data, (iii) one geophysicist to label $7636$ faults in the region. Note that all novices, practitioners, and the expert segment faults on the same subsurface volume with disagreements between and among the novices and practitioners. Additionally, each fault annotation is equipped with the confidence level of the annotator. The paper provides benchmarks on detecting and segmenting the expert labels, given the novice and practitioner labels. Additional details along with the dataset links and codes are available at $\href{https://alregib.ece.gatech.edu/cracks-crowdsourcing-resources-for-analysis-and-categorization-of-key-subsurface-faults/}{link}$.