Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization

TL;DR

This paper introduces a kernel-based adversarial training method that efficiently solves inner maximization problems, adapts to noise and smoothness, and demonstrates strong empirical performance in robust learning.

Contribution

It presents a novel feature-space adversarial training formulation in RKHS that is computationally efficient and adaptable, with theoretical guarantees and extensions to multiple kernel learning.

Findings

Exact solution of inner maximization problem

Efficient optimization via kernel ridge regression

Strong empirical performance in adversarial settings

Abstract

Adversarial training has emerged as a key technique to enhance model robustness against adversarial input perturbations. Many of the existing methods rely on computationally expensive min-max problems that limit their application in practice. We propose a novel formulation of adversarial training in reproducing kernel Hilbert spaces, shifting from input to feature-space perturbations. This reformulation enables the exact solution of inner maximization and efficient optimization. It also provides a regularized estimator that naturally adapts to the noise level and the smoothness of the underlying function. We establish conditions under which the feature-perturbed formulation is a relaxation of the original problem and propose an efficient optimization algorithm based on iterative kernel ridge regression. We provide generalization bounds that help to understand the properties of the method. We also extend the formulation to multiple kernel learning. Empirical evaluation shows good performance in both clean and adversarial settings.