Can You Explain That, Better? Comprehensible Text Analytics for SE Applications

TL;DR

This paper demonstrates that combining LDA with FFTs creates simple, human-readable models for classifying software bug reports, achieving comparable or better accuracy than complex methods while being faster and easier to interpret.

Contribution

It introduces a novel approach using LDA and FFTs for software bug report classification, emphasizing simplicity and interpretability over complexity.

Findings

LDA+FFTs models are small and human-readable.

LDA+FFTs achieve similar or better accuracy than complex models.

LDA+FFTs are faster to generate and easier to understand.

Abstract

Text mining methods are used for a wide range of Software Engineering (SE) tasks. The biggest challenge of text mining is high dimensional data, i.e., a corpus of documents can contain $10^4$ to $10^6$ unique words. To address this complexity, some very convoluted text mining methods have been applied. Is that complexity necessary? Are there simpler ways to quickly generate models that perform as well as the more convoluted methods and also be human-readable? To answer these questions, we explore a combination of LDA (Latent Dirichlet Allocation) and FFTs (Fast and Frugal Trees) to classify NASA software bug reports from six different projects. Designed using principles from psychological science, FFTs return very small models that are human-comprehensible. When compared to the commonly used text mining method and a recent state-of-the-art-system (search-based SE method that automatically tune the control parameters of LDA), these FFT models are very small (a binary tree of depth $d = 4$ that references only 4 topics) and hence easy to understand. They were also faster to generate and produced similar or better severity predictions. Hence we can conclude that, at least for datasets explored here, convoluted text mining models can be deprecated in favor of simpler method such as LDA+FFTs. At the very least, we recommend LDA+FFTs (a) when humans need to read, understand, and audit a model or (b) as an initial baseline method for the SE researchers exploring text artifacts from software projects.