Disagreement-Regularized Importance Sampling for Adversarial Label Corruption

TL;DR

This paper introduces Disagreement-Regularized Importance Sampling (DR-IS), a novel method that enhances robustness against adversarial label corruption by leveraging loss rank disagreement, with theoretical guarantees and empirical validation.

Contribution

The paper proposes DR-IS, a new subsampling technique based on rank disagreement, with proven concentration bounds and improved robustness over existing methods.

Findings

DR-IS maintains robustness under targeted high-norm attacks.

Theoretical bounds certify strict separation between clean and corrupted examples.

Empirical results show DR-IS outperforms magnitude-based methods like EL2N.

Abstract

Standard Importance Sampling (IS) collapses under label corruption because high-norm examples, prioritized for variance reduction, are often adversarial outliers. We formalize this misalignment using an $\varepsilon$-contamination model and propose Disagreement-Regularized Importance Sampling (DR-IS), a sub-sampling method based on loss rank-disagreement across independent proxy ensemble. We prove finite-sample concentration bounds showing that the empirical rank disagreement of bulk corrupted examples is bounded above, and that of boundary-clean examples bounded below, both at rate $O(\sqrt{\log(N/\delta)/K})$ with probability $1-\delta$; when the structural expectation gap $\Delta'$ between the two groups is positive and the boundary-clean set is at least as large as the selected subset, these bounds certify strict separation and control the contamination rate of the selected subset. Empirically, DR-IS remains robust under targeted high-norm attacks that break magnitude-based methods such as the Error $L_2$-norm (EL2N) on benchmark datasets. DR-IS complements training-dynamics approaches like Area Under the Margin ranking (AUM), offering improved robustness in the loss-aligned regime alongside explicit finite-sample concentration certificates and a contamination bound limiting noise leakage from the statistical tail of corrupted points.