Bennett-type Generalization Bounds: Large-deviation Case and Faster Rate of Convergence

TL;DR

This paper introduces Bennett-type generalization bounds for i.i.d. samples that converge faster than traditional bounds, especially in large-deviation scenarios, enhancing understanding of learning process convergence.

Contribution

The paper develops new Bennett-type deviation inequalities and bounds with faster convergence rates, particularly in large-deviation cases, improving upon traditional generalization bounds.

Findings

Bounds have a convergence rate of o(N^{-1/2})

Faster bounds in large-deviation scenarios

Comparison with existing asymptotic results

Abstract

In this paper, we present the Bennett-type generalization bounds of the learning process for i.i.d. samples, and then show that the generalization bounds have a faster rate of convergence than the traditional results. In particular, we first develop two types of Bennett-type deviation inequality for the i.i.d. learning process: one provides the generalization bounds based on the uniform entropy number; the other leads to the bounds based on the Rademacher complexity. We then adopt a new method to obtain the alternative expressions of the Bennett-type generalization bounds, which imply that the bounds have a faster rate o(N^{-1/2}) of convergence than the traditional results O(N^{-1/2}). Additionally, we find that the rate of the bounds will become faster in the large-deviation case, which refers to a situation where the empirical risk is far away from (at least not close to) the expected risk. Finally, we analyze the asymptotical convergence of the learning process and compare our analysis with the existing results.