Stability of the Stochastic Gradient Method for an Approximated Large Scale Kernel Machine

TL;DR

This paper analyzes the stability and generalization performance of stochastic gradient methods when used with approximated kernel functions via random Fourier features, demonstrating theoretical stability and empirical validation.

Contribution

It provides a theoretical analysis of the stability of stochastic gradient methods for approximated kernel machines and empirically verifies the results across multiple datasets.

Findings

SGM is stable for approximated kernel functions under certain conditions.

High probability bounds on generalization error are established.

Empirical results confirm theoretical stability and generalization performance.

Abstract

In this paper we measured the stability of stochastic gradient method (SGM) for learning an approximated Fourier primal support vector machine. The stability of an algorithm is considered by measuring the generalization error in terms of the absolute difference between the test and the training error. Our problem is to learn an approximated kernel function using random Fourier features for a binary classification problem via online convex optimization settings. For a convex, Lipschitz continuous and smooth loss function, given reasonable number of iterations stochastic gradient method is stable. We showed that with a high probability SGM generalizes well for an approximated kernel under given assumptions.We empirically verified the theoretical findings for different parameters using several data sets.