f-divergences and their applications in lossy compression and bounding generalization error

TL;DR

This paper explores the use of $f$-divergences in three areas: improving tail probability bounds, enhancing finite blocklength lossy compression bounds, and bounding generalization error in learning algorithms, offering new theoretical insights.

Contribution

It introduces novel bounds and connections involving $f$-divergences, improving existing results in tail probability estimates, lossy compression, and generalization error analysis.

Findings

Generalized Sanov's bound with super-modular $f$-divergence improves over classical bounds.

New bounds on achievable rates in finite blocklength lossy compression using mutual $f$-information.

A novel connection between $f$-divergences and generalization error bounds in machine learning.

Abstract

In this paper, we provide three applications for $f$-divergences: (i) we introduce Sanov's upper bound on the tail probability of the sum of independent random variables based on super-modular $f$-divergence and show that our generalized Sanov's bound strictly improves over ordinary one, (ii) we consider the lossy compression problem which studies the set of achievable rates for a given distortion and code length. We extend the rate-distortion function using mutual $f$-information and provide new and strictly better bounds on achievable rates in the finite blocklength regime using super-modular $f$-divergences, and (iii) we provide a connection between the generalization error of algorithms with bounded input/output mutual $f$-information and a generalized rate-distortion problem. This connection allows us to bound the generalization error of learning algorithms using lower bounds on the $f$-rate-distortion function. Our bound is based on a new lower bound on the rate-distortion function that (for some examples) strictly improves over previously best-known bounds.