Testing Framework for Black-box AI Models

TL;DR

This paper introduces a comprehensive testing framework for black-box AI models that automates test generation across multiple data types and properties, improving reliability and uncovering issues in industrial applications.

Contribution

It presents a novel end-to-end generic testing framework capable of automated testing for various AI modalities and properties, enhancing model validation processes.

Findings

Effective in uncovering issues in industrial AI models

Supports multiple data modalities and properties

Demonstrated success through a practical demo

Abstract

With widespread adoption of AI models for important decision making, ensuring reliability of such models remains an important challenge. In this paper, we present an end-to-end generic framework for testing AI Models which performs automated test generation for different modalities such as text, tabular, and time-series data and across various properties such as accuracy, fairness, and robustness. Our tool has been used for testing industrial AI models and was very effective to uncover issues present in those models. Demo video link: https://youtu.be/984UCU17YZI