Deviation bound for non-causal machine learning

TL;DR

This paper introduces a new concentration inequality framework for non-causal data in machine learning, particularly addressing challenges in deep neural networks used in natural language processing.

Contribution

It provides a Hoeffding-type concentration inequality for non-causal random fields, expanding the applicability of concentration bounds to non-causal data.

Findings

Established a Hoeffding-type inequality for non-causal random fields

Provided a local approximation method for non-causal data

Extended concentration inequality applicability to deep learning models

Abstract

Concentration inequalities are widely used for analyzing machine learning algorithms. However, current concentration inequalities cannot be applied to some of the most popular deep neural networks, notably in natural language processing. This is mostly due to the non-causal nature of such involved data, in the sense that each data point depends on other neighbor data points. In this paper, a framework for modeling non-causal random fields is provided and a Hoeffding-type concentration inequality is obtained for this framework. The proof of this result relies on a local approximation of the non-causal random field by a function of a finite number of i.i.d. random variables.