General Frameworks for Conditional Two-Sample Testing

TL;DR

This paper introduces two general frameworks for conditional two-sample testing, addressing the challenge of comparing distributions while controlling for confounders, with theoretical and practical insights.

Contribution

It proposes novel methods to convert conditional independence tests into two-sample tests and to compare marginal distributions using density ratios.

Findings

Frameworks enable valid conditional two-sample testing under certain assumptions

Conversion preserves asymptotic properties of the original tests

Simulation studies demonstrate effectiveness in finite samples

Abstract

We study the problem of conditional two-sample testing, which aims to determine whether two populations have the same distribution after accounting for confounding factors. This problem commonly arises in various applications, such as domain adaptation and algorithmic fairness, where comparing two groups is essential while controlling for confounding variables. We begin by establishing a hardness result for conditional two-sample testing, demonstrating that no valid test can have significant power against any single alternative without proper assumptions. We then introduce two general frameworks that implicitly or explicitly target specific classes of distributions for their validity and power. Our first framework allows us to convert any conditional independence test into a conditional two-sample test in a black-box manner, while preserving the asymptotic properties of the original conditional independence test. The second framework transforms the problem into comparing marginal distributions with estimated density ratios, which allows us to leverage existing methods for marginal two-sample testing. We demonstrate this idea in a concrete manner with classification and kernel-based methods. Finally, simulation studies are conducted to illustrate the proposed frameworks in finite-sample scenarios.