On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspective

TL;DR

This paper provides a rigorous convergence proof for over-parameterized Variational Autoencoders using Neural Tangent Kernel techniques, linking their optimization dynamics to Kernel Ridge Regression and validating findings through experiments.

Contribution

It introduces a novel NTK-based analysis of VAE convergence and establishes a connection to Kernel Ridge Regression, advancing theoretical understanding of generative model training.

Findings

Proves convergence of over-parameterized VAEs under mild assumptions

Links VAE optimization to Kernel Ridge Regression

Validates theoretical results with experimental simulations

Abstract

Variational Auto-Encoders (VAEs) have emerged as powerful probabilistic models for generative tasks. However, their convergence properties have not been rigorously proven. The challenge of proving convergence is inherently difficult due to the highly non-convex nature of the training objective and the implementation of a Stochastic Neural Network (SNN) within VAE architectures. This paper addresses these challenges by characterizing the optimization trajectory of SNNs utilized in VAEs through the lens of Neural Tangent Kernel (NTK) techniques. These techniques govern the optimization and generalization behaviors of ultra-wide neural networks. We provide a mathematical proof of VAE convergence under mild assumptions, thus advancing the theoretical understanding of VAE optimization dynamics. Furthermore, we establish a novel connection between the optimization problem faced by over-parameterized SNNs and the Kernel Ridge Regression (KRR) problem. Our findings not only contribute to the theoretical foundation of VAEs but also open new avenues for investigating the optimization of generative models using advanced kernel methods. Our theoretical claims are verified by experimental simulations.