An empirical evaluation of the usefulness of Tree Kernels for Commit-time Defect Detection in large software systems

TL;DR

This paper evaluates the effectiveness of tree kernel methods at commit time for detecting bugs in large software systems, focusing on method-level analysis and comparing with existing clone detection tools.

Contribution

It introduces a tree kernel-based approach for commit-time defect detection that considers method-level changes and compares its performance with NiCad on benchmark datasets.

Findings

Tree kernels achieve comparable clone detection performance to NiCad.

The approach effectively detects defect-inducing commits with high accuracy.

Method-level analysis provides targeted insights for developers.

Abstract

Defect detection at commit check-in time prevents the introduction of defects into software systems. Current defect detection approaches rely on metric-based models which are not very accurate and whose results are not directly useful for developers. We propose a method to detect bug-inducing commits by comparing the incoming changes with all past commits in the project, considering both those that introduced defects and those that did not. Our method considers individual changes in the commit separately, at the method-level granularity. Doing so helps developers as they are informed of specific methods that need further attention instead of being told that the entire commit is problematic. Our approach represents source code as abstract syntax trees and uses tree kernels to estimate the similarity of the code with previous commits. We experiment with subtree kernels (STK), subset tree kernels (SSTK), or partial tree kernels (PTK). An incoming change is then classified using a K-NN classifier on the past changes. We evaluate our approach on the BigCloneBench benchmark and on the Technical Debt dataset, using the NiCad clone detector as the baseline. Our experiments with the BigCloneBench benchmark show that the tree kernel approach can detect clones with a comparable MAP to that of NiCad. Also, on defect detection with the Technical Debt dataset, tree kernels are least as effective as NiCad with MRR, F-score, and Accuracy of 0.87, 0.80, and 0.82 respectively.