Regularized $f$-Divergence Kernel Tests

TL;DR

This paper introduces a flexible kernel-based framework for two-sample testing using $f$-divergences, with theoretical guarantees and applications to privacy auditing and machine unlearning evaluation.

Contribution

It develops a practical, adaptive kernel test for $f$-divergences, including the Hockey-Stick divergence, with theoretical power guarantees and diverse application scenarios.

Findings

Different $f$-divergences detect localized distribution differences.

The proposed test is adaptive over hyperparameters.

Applications include differential privacy auditing and machine unlearning.

Abstract

We propose a framework to construct practical kernel-based two-sample tests from the family of $f$-divergences. The test statistic is computed from the witness function of a regularized variational representation of the divergence, which we estimate using kernel methods. The proposed test is adaptive over hyperparameters such as the kernel bandwidth and the regularization parameter. We provide theoretical guarantees for statistical test power across our family of $f$-divergence estimates. While our test covers a variety of $f$-divergences, we bring particular focus to the Hockey-Stick divergence, motivated by its applications to differential privacy auditing and machine unlearning evaluation. For two-sample testing, experiments demonstrate that different $f$-divergences are sensitive to different localized differences, illustrating the importance of leveraging diverse statistics. For machine unlearning, we propose a relative test that distinguishes true unlearning failures from safe distributional variations.