A Defect Classification Framework for AI-Based Software Systems (AI-ODC)

TL;DR

This paper introduces a modified defect classification framework tailored for AI-based software systems, capturing their unique attributes and aiding in targeted defect analysis and quality assurance.

Contribution

It adapts the Orthogonal Defect Classification (ODC) model specifically for AI systems by adding new attributes and severity levels, enabling effective defect analysis of AI-specific characteristics.

Findings

Defects during the Learning phase are most prevalent and high severity.

Defects in the Thinking phase significantly impact trustworthiness and accuracy.

The AI-ODC framework effectively identifies high-risk defect categories.

Abstract

Artificial Intelligence has gained a lot of attention recently, it has been utilized in several fields ranging from daily life activities, such as responding to emails and scheduling appointments, to manufacturing and automating work activities. Artificial Intelligence systems are mainly implemented as software solutions, and it is essential to discover and remove software defects to assure its quality using defect analysis which is one of the major activities that contribute to software quality. Despite the proliferation of AI-based systems, current defect analysis models fail to capture their unique attributes. This paper proposes a framework inspired by the Orthogonal Defect Classification (ODC) paradigm and enables defect analysis of Artificial Intelligence systems while recognizing its special attributes and characteristics. This study demonstrated the feasibility of modifying ODC for AI systems to classify its defects. The ODC was adjusted to accommodate the Data, Learning, and Thinking aspects of AI systems which are newly introduced classification dimensions. This adjustment involved the introduction of an additional attribute to the ODC attributes, the incorporation of a new severity level, and the substitution of impact areas with characteristics pertinent to AI systems. The framework was showcased by applying it to a publicly available Machine Learning bug dataset, with results analyzed through one-way and two-way analysis. The case study indicated that defects occurring during the Learning phase were the most prevalent and were significantly linked to high-severity classifications. In contrast, defects identified in the Thinking phase had a disproportionate effect on trustworthiness and accuracy. These findings illustrate AIODC's capability to identify high-risk defect categories and inform focused quality assurance measures.