Generalization Error Bounds on Deep Learning with Markov Datasets

TL;DR

This paper establishes upper bounds on the generalization error of deep neural networks trained on Markov datasets, extending existing bounds to non-i.i.d. data using new probabilistic inequalities.

Contribution

It introduces novel symmetrization inequalities for Markov chains and adapts generalization bounds to Markov and Bayesian data settings in deep learning.

Findings

Derived bounds depend on the spectral gap of the Markov chain.

Extended bounds to AR, ARMA, and mixture models.

Proposed a method to convert traditional bounds to Bayesian counterparts.

Abstract

In this paper, we derive upper bounds on generalization errors for deep neural networks with Markov datasets. These bounds are developed based on Koltchinskii and Panchenko's approach for bounding the generalization error of combined classifiers with i.i.d. datasets. The development of new symmetrization inequalities in high-dimensional probability for Markov chains is a key element in our extension, where the spectral gap of the infinitesimal generator of the Markov chain plays a key parameter in these inequalities. We also propose a simple method to convert these bounds and other similar ones in traditional deep learning and machine learning to Bayesian counterparts for both i.i.d. and Markov datasets. Extensions to $m$-order homogeneous Markov chains such as AR and ARMA models and mixtures of several Markov data services are given.