K-Detector: Identifying Duplicate Crash Failures in Large-Scale Software Delivery

TL;DR

K-Detector is a machine learning-based tool designed to automatically identify duplicate crash failures in large-scale software delivery, significantly reducing manual effort and improving efficiency.

Contribution

It introduces a novel training-based mathematical model utilizing component information for crash similarity comparison and demonstrates its effectiveness in real-world deployment.

Findings

Achieved 0.986 AUC in crash duplicate detection

Saved 97% human effort in crash triage

Validated on 11,208 samples

Abstract

After a developer submits code, corresponding test cases arise to ensure the quality of software delivery. Test failures would occur during this period, such as crash, error, and timeout. Since it takes time for developers to resolve them, many duplicate failures will happen during this period. In the delivery practice of SAP HANA, crash triage is considered as the most time-consuming task. If duplicate crash failures can be automatically identified, the degree of automation will be significantly enhanced. To find such duplicates, we propose a training-based mathematical model that utilizes component information of SAP HANA to achieve better crash similarity comparison. We implement our approach in a tool named Knowledge-based Detector (K-Detector), which is verified by 11,208 samples and performs 0.986 in AUC. Furthermore, we have deployed K-Detector to the production environment, and it can save 97% human efforts in crash triage as statistics.