Complex SGD and Directional Bias in Reproducing Kernel Hilbert Spaces

TL;DR

This paper introduces a complex variant of SGD with convergence guarantees and demonstrates its effectiveness in kernel regression problems involving complex reproducing kernel Hilbert spaces.

Contribution

It proposes a complex SGD algorithm with proven convergence under real setting assumptions and extends directional bias results to complex kernel regression.

Findings

Complex SGD converges under similar assumptions as real SGD.

Empirical results show effectiveness in recovering superoscillation functions.

Demonstrates recovery of Blaschke products in Hardy Space.

Abstract

Stochastic Gradient Descent (SGD) is a known stochastic iterative method popular for large-scale convex optimization problems due to its simple implementation and scalability. Some objectives, such as those found in complex-valued neural networks, benefit from updates like in SGD and Gradient Descent (GD) with a newly defined ``gradient'' that allows for complex parameters. This complex variant of the SGD/GD methods has already been proposed, but convergence guarantees without analyticity constraints have not yet been provided. We propose a variant of SGD (complex SGD) that allows for complex parameters, and we provide convergence guarantees under assumptions that parallel those from the real setting. Notably, these results extend to GD as well, and with the same set of assumptions, we confirm that some directional bias results extend from the real to the complex setting for kernel regression problems. We provide empirical results demonstrating the efficacy of the complex SGD in kernel regression problems utilizing complex reproducing kernel Hilbert spaces. In particular, we demonstrate we may recover superoscillation functions and Blaschke products from the Fock Space and Hardy Space, respectively, as the optimal functions for a particular choice of a loss function.