Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data

TL;DR

This paper introduces meta two-sample testing (M2ST), leveraging auxiliary data to learn kernels that enable effective testing on new tasks with limited data, outperforming traditional methods.

Contribution

It proposes novel meta-learning algorithms for kernel-based two-sample tests that work well with scarce data, supported by theoretical and empirical validation.

Findings

Meta testing schemes outperform direct kernel learning in limited data scenarios.

The tailored approach achieves higher test power than generic schemes.

Theoretical analysis identifies conditions for successful meta-testing.

Abstract

Modern kernel-based two-sample tests have shown great success in distinguishing complex, high-dimensional distributions with appropriate learned kernels. Previous work has demonstrated that this kernel learning procedure succeeds, assuming a considerable number of observed samples from each distribution. In realistic scenarios with very limited numbers of data samples, however, it can be challenging to identify a kernel powerful enough to distinguish complex distributions. We address this issue by introducing the problem of meta two-sample testing (M2ST), which aims to exploit (abundant) auxiliary data on related tasks to find an algorithm that can quickly identify a powerful test on new target tasks. We propose two specific algorithms for this task: a generic scheme which improves over baselines and a more tailored approach which performs even better. We provide both theoretical justification and empirical evidence that our proposed meta-testing schemes out-perform learning kernel-based tests directly from scarce observations, and identify when such schemes will be successful.