Improved Generalization Bounds for Transductive Learning by Transductive Local Complexity and Its Applications

TL;DR

This paper introduces Transductive Local Complexity (TLC), a novel framework extending local Rademacher complexity to transductive learning, providing sharp generalization bounds and advancing theoretical understanding in this area.

Contribution

The paper develops TLC, a new complexity measure for transductive learning, and derives nearly sharp excess risk bounds, addressing open problems and improving existing bounds.

Findings

Derived a new concentration inequality for the test-train process.

Established nearly optimal bounds for realizable transductive learning with finite VC dimension.

Provided a sharper excess risk bound for transductive kernel learning.

Abstract

We introduce Transductive Local Complexity (TLC) to extend the classical Local Rademacher Complexity (LRC) to the transductive setting, incorporating substantial and novel components. Although LRC has been used to obtain sharp generalization bounds and minimax rates for inductive tasks such as classification and nonparametric regression, it has remained an open problem whether a localized Rademacher complexity framework can be effectively adapted to transductive learning to achieve sharp or nearly sharp bounds consistent with inductive results. We provide an affirmative answer via TLC. TLC is constructed by first deriving a new concentration inequality in Theorem 4.1 for the supremum of empirical processes capturing the gap between test and training losses, termed the test-train process, under uniform sampling without replacement, which leverages a novel combinatorial property of the test-train process and a new proof strategy applying the exponential Efron-Stein inequality twice. A subsequent peeling strategy applied to a new decomposition of the expectation of the test-train process and a new surrogate variance operator then yield excess risk bounds in the transductive setting that are nearly consistent with classical LRC-based inductive bounds up to a logarithmic gap. We further advance transductive learning through two applications: (1) for realizable transductive learning over binary-valued classes with finite VC dimension of $\dVC$ and $u \ge m \ge \dVC$, where $u$ and $m$ are the number of test features and training features, our Theorem 6.1 gives a nearly optimal bound $\Theta(\dVC \log(me/\dVC)/m)$ matching the minimax rate $\Theta(\dVC/m)$ up to $\log m$, resolving a decade-old open question; and (2) Theorem 6.2 presents a sharper excess risk bound for transductive kernel learning compared to the current state-of-the-art.