Kernel Selection is Model Selection: A Unified Complexity-Penalized Approach for MMD Two-Sample Tests

TL;DR

This paper introduces CP-MMD, a new complexity-penalized criterion for kernel selection in MMD two-sample tests, enabling continuous, grid-free optimization that maximizes test power while maintaining Type-I error control.

Contribution

It formulates kernel selection as a model selection problem and develops CP-MMD, which accounts for optimization complexity and supports continuous kernel parameter search.

Findings

CP-MMD achieves higher or comparable test power to state-of-the-art methods.

It enables grid-free kernel selection over continuous parameter spaces.

CP-MMD maintains unconditional Type-I error control.

Abstract

The Maximum Mean Discrepancy (MMD) is a cornerstone statistic for nonparametric two-sample testing, but its test power is dictated entirely by the chosen kernel. Because any fixed kernel inherently fails to distinguish certain distributions, the kernel must be dynamically optimized. However, data-driven optimization violates the foundational i.i.d. assumption, forcing a strict trade-off in existing frameworks. Ratio criteria ignore this dependence, inducing overfitting and variance collapse on rich kernel classes. Conversely, aggregation methods bypass the dependence using finite grids, but this strategy cannot scale to continuous search spaces like deep kernels. To break this dichotomy, we establish data-driven kernel selection as a model selection problem. We propose Complexity-Penalized MMD (CP-MMD), a criterion derived by applying the two-sample uniform concentration inequality of preceding works to the post-optimization MMD problem. The resulting penalty bounds the empirical MMD by the complexity of the kernel search space, mathematically absorbing the cost of optimization, so that CP-MMD enables direct, grid-free maximization over continuous parametric classes, including scalar bandwidths, polynomial feature bandwidths, and deep network parameters. By formally accounting for optimization complexity, we prove that CP-MMD maximizes true test power while ensuring unconditional Type-I validity. Consequently, CP-MMD enables grid-free kernel selection across linear, polynomial-feature, and deep regimes, matching or exceeding state-of-the-art test power.