Multi-Label Requirements Classification with Large Taxonomies

TL;DR

This paper explores zero-shot learning for multi-label requirements classification with large taxonomies, evaluating classifier types, hierarchy effects, and taxonomy structure on performance, aiming to improve requirements traceability.

Contribution

It systematically investigates zero-shot learning for large taxonomy multi-label classification in requirements engineering, including a ground truth creation process and analysis of classification strategies.

Findings

Sentence-based classifiers have higher recall than word-based.

Hierarchical classification does not always improve performance.

Taxonomy size negatively correlates with recall in hierarchical classifiers.

Abstract

Classification aids software development activities by organizing requirements in classes for easier access and retrieval. The majority of requirements classification research has, so far, focused on binary or multi-class classification. Multi-label classification with large taxonomies could aid requirements traceability but is prohibitively costly with supervised training. Hence, we investigate zero-short learning to evaluate the feasibility of multi-label requirements classification with large taxonomies. We associated, together with domain experts from the industry, 129 requirements with 769 labels from taxonomies ranging between 250 and 1183 classes. Then, we conducted a controlled experiment to study the impact of the type of classifier, the hierarchy, and the structural characteristics of taxonomies on the classification performance. The results show that: (1) The sentence-based classifier had a significantly higher recall compared to the word-based classifier; however, the precision and F1-score did not improve significantly. (2) The hierarchical classification strategy did not always improve the performance of requirements classification. (3) The total and leaf nodes of the taxonomies have a strong negative correlation with the recall of the hierarchical sentence-based classifier. We investigate the problem of multi-label requirements classification with large taxonomies, illustrate a systematic process to create a ground truth involving industry participants, and provide an analysis of different classification pipelines using zero-shot learning.