Explicit Learning Curves for Transduction and Application to Clustering and Compression Algorithms

TL;DR

This paper develops explicit learning curves for transduction, extending Vapnik's bounds, and applies them to clustering and compression algorithms, providing new theoretical insights and error bounds.

Contribution

It introduces an explicit PAC-Bayesian transductive bound and applies it to derive error bounds for transductive SVMs and clustering algorithms.

Findings

Derived explicit transductive error bounds.

Extended Vapnik's transductive bounds using concentration inequalities.

Applied bounds to support vector machines and clustering algorithms.

Abstract

Inductive learning is based on inferring a general rule from a finite data set and using it to label new data. In transduction one attempts to solve the problem of using a labeled training set to label a set of unlabeled points, which are given to the learner prior to learning. Although transduction seems at the outset to be an easier task than induction, there have not been many provably useful algorithms for transduction. Moreover, the precise relation between induction and transduction has not yet been determined. The main theoretical developments related to transduction were presented by Vapnik more than twenty years ago. One of Vapnik's basic results is a rather tight error bound for transductive classification based on an exact computation of the hypergeometric tail. While tight, this bound is given implicitly via a computational routine. Our first contribution is a somewhat looser but explicit characterization of a slightly extended PAC-Bayesian version of Vapnik's transductive bound. This characterization is obtained using concentration inequalities for the tail of sums of random variables obtained by sampling without replacement. We then derive error bounds for compression schemes such as (transductive) support vector machines and for transduction algorithms based on clustering. The main observation used for deriving these new error bounds and algorithms is that the unlabeled test points, which in the transductive setting are known in advance, can be used in order to construct useful data dependent prior distributions over the hypothesis space.