METTLE: a METamorphic testing approach to assessing and validating unsupervised machine LEarning systems

TL;DR

This paper introduces METTLE, a metamorphic testing approach for assessing and validating unsupervised machine learning systems based on user expectations, with experiments demonstrating its effectiveness in selecting suitable clustering algorithms.

Contribution

The paper presents a novel METTLE framework with 11 metamorphic relations tailored for unsupervised learning validation, emphasizing user-specific requirements and providing a practical assessment method.

Findings

End users can effectively assess clustering systems using MR-based criteria.

METTLE reveals system behaviors aligned with user expectations.

The approach aids in selecting appropriate unsupervised learning methods.

Abstract

Unsupervised machine learning is the training of an artificial intelligence system using information that is neither classified nor labeled, with a view to modeling the underlying structure or distribution in a dataset. Since unsupervised machine learning systems are widely used in many real-world applications, assessing the appropriateness of these systems and validating their implementations with respect to individual users' requirements and specific application scenarios$\,/\,$contexts are indisputably two important tasks. Such assessment and validation tasks, however, are fairly challenging due to the absence of a priori knowledge of the data. In view of this challenge, we develop a $\textbf{MET}$amorphic $\textbf{T}$esting approach to assessing and validating unsupervised machine $\textbf{LE}$arning systems, abbreviated as METTLE. Our approach provides a new way to unveil the (possibly latent) characteristics of various machine learning systems, by explicitly considering the specific expectations and requirements of these systems from individual users' perspectives. To support METTLE, we have further formulated 11 generic metamorphic relations (MRs), covering users' generally expected characteristics that should be possessed by machine learning systems. To demonstrate the viability and effectiveness of METTLE we have performed an experiment involving six commonly used clustering systems. Our experiment has shown that, guided by user-defined MR-based adequacy criteria, end users are able to assess, validate, and select appropriate clustering systems in accordance with their own specific needs. Our investigation has also yielded insightful understanding and interpretation of the behavior of the machine learning systems from an end-user software engineering's perspective, rather than a designer's or implementor's perspective, who normally adopts a theoretical approach.