MLCheck- Property-Driven Testing of Machine Learning Models

TL;DR

MLCheck introduces a property-driven testing framework for machine learning models, enabling systematic validation across diverse requirements and models with competitive performance.

Contribution

It provides a novel property specification language and a test case generation technique applicable to various ML models and requirements.

Findings

MLCheck effectively tests models for fairness, knowledge graph learning, and security.

It outperforms specialized testing methods in some scenarios.

The approach maintains comparable runtime to existing methods.

Abstract

In recent years, we observe an increasing amount of software with machine learning components being deployed. This poses the question of quality assurance for such components: how can we validate whether specified requirements are fulfilled by a machine learned software? Current testing and verification approaches either focus on a single requirement (e.g., fairness) or specialize on a single type of machine learning model (e.g., neural networks). In this paper, we propose property-driven testing of machine learning models. Our approach MLCheck encompasses (1) a language for property specification, and (2) a technique for systematic test case generation. The specification language is comparable to property-based testing languages. Test case generation employs advanced verification technology for a systematic, property-dependent construction of test suites, without additional user-supplied generator functions. We evaluate MLCheck using requirements and data sets from three different application areas (software discrimination, learning on knowledge graphs and security). Our evaluation shows that despite its generality MLCheck can even outperform specialised testing approaches while having a comparable runtime.