Credal Two-Sample Tests of Epistemic Uncertainty

TL;DR

This paper introduces credal two-sample tests that compare sets of probability measures to incorporate epistemic uncertainty, providing a more flexible and robust framework for hypothesis testing under partial ignorance.

Contribution

It presents the first nonparametric credal two-sample testing framework, including permutation-based solutions and kernel methods for practical applications.

Findings

Enables reasoning about equality, inclusion, intersection, and mutual exclusivity of credal sets.

Provides a permutation-based method improving robustness of credal set comparisons.

Incorporates epistemic uncertainty into hypothesis testing for more credible conclusions.

Abstract

We introduce credal two-sample testing, a new hypothesis testing framework for comparing credal sets -- convex sets of probability measures where each element captures aleatoric uncertainty and the set itself represents epistemic uncertainty that arises from the modeller's partial ignorance. Compared to classical two-sample tests, which focus on comparing precise distributions, the proposed framework provides a broader and more versatile set of hypotheses. This approach enables the direct integration of epistemic uncertainty, effectively addressing the challenges arising from partial ignorance in hypothesis testing. By generalising two-sample test to compare credal sets, our framework enables reasoning for equality, inclusion, intersection, and mutual exclusivity, each offering unique insights into the modeller's epistemic beliefs. As the first work on nonparametric hypothesis testing for comparing credal sets, we focus on finitely generated credal sets derived from i.i.d. samples from multiple distributions -- referred to as credal samples. We formalise these tests as two-sample tests with nuisance parameters and introduce the first permutation-based solution for this class of problems, significantly improving existing methods. Our approach properly incorporates the modeller's epistemic uncertainty into hypothesis testing, leading to more robust and credible conclusions, with kernel-based implementations for real-world applications.