Statistical Confidence in Functional Correctness: An Approach for AI Product Functional Correctness Evaluation

TL;DR

This paper introduces a statistically robust approach called SCFC for evaluating AI systems' functional correctness, linking business requirements to confidence measures that account for performance variability.

Contribution

It presents a novel method for assessing AI functional correctness using statistical confidence, bridging the gap between theoretical standards and practical evaluation.

Findings

The SCFC approach provides a feasible way to quantify confidence in AI correctness.

Case studies show the approach's utility and ease of use in industry.

Experts find the method valuable for practical AI quality assessment.

Abstract

The quality assessment of Artificial Intelligence (AI) systems is a fundamental challenge due to their inherently probabilistic nature. Standards such as ISO/IEC 25059 provide a quality model, but they lack practical and statistically robust methods for assessing functional correctness. This paper proposes and evaluates the Statistical Confidence in Functional Correctness (SCFC) approach, which seeks to fill this gap by connecting business requirements to a measure of statistical confidence that considers both the model's average performance and its variability. The approach consists of four steps: defining quantitative specification limits, performing stratified and probabilistic sampling, applying bootstrapping to estimate a confidence interval for the performance metric, and calculating a capability index as a final indicator. The approach was evaluated through a case study on two real-world AI systems in industry involving interviews with AI experts. Valuable insights were collected from the experts regarding the utility, ease of use, and intention to adopt the methodology in practical scenarios. We conclude that the proposed approach is a feasible and valuable way to operationalize the assessment of functional correctness, moving the evaluation from a point estimate to a statement of statistical confidence.