A Mean-Field Theory for Kernel Alignment with Random Features in Generative and Discriminative Models

TL;DR

This paper introduces a mean-field theory-based supervised kernel learning method for generative and discriminative models, improving kernel optimization in MMD GANs and SVMs through a distributionally robust approach and particle SGD.

Contribution

It develops a novel distributionally robust kernel optimization framework using mean-field analysis and particle SGD, with proven convergence and improved hypothesis testing power.

Findings

Enhanced kernel MMD test power for larger thresholds.

Empirical results show superior performance over existing kernel learning methods.

Theoretical proof of convergence and consistency of the proposed method.

Abstract

We propose a novel supervised learning method to optimize the kernel in the maximum mean discrepancy generative adversarial networks (MMD GANs), and the kernel support vector machines (SVMs). Specifically, we characterize a distributionally robust optimization problem to compute a good distribution for the random feature model of Rahimi and Recht. Due to the fact that the distributional optimization is infinite dimensional, we consider a Monte-Carlo sample average approximation (SAA) to obtain a more tractable finite dimensional optimization problem. We subsequently leverage a particle stochastic gradient descent (SGD) method to solve the derived finite dimensional optimization problem. Based on a mean-field analysis, we then prove that the empirical distribution of the interactive particles system at each iteration of the SGD follows the path of the gradient descent flow on the Wasserstein manifold. We also establish the non-asymptotic consistency of the finite sample estimator. We evaluate our kernel learning method for the hypothesis testing problem by evaluating the kernel MMD statistics, and show that our learning method indeed attains better power of the test for larger threshold values compared to an untrained kernel. Moreover, our empirical evaluation on benchmark data-sets shows the advantage of our kernel learning approach compared to alternative kernel learning methods.