Hedging predictions in machine learning

TL;DR

This paper introduces a new hedging technique for machine learning predictions that provides reliable confidence measures and error control, applicable to various algorithms like SVMs and kernel methods.

Contribution

It presents a novel method for producing confidence-aware predictions that are statistically valid and adaptable to multiple machine learning algorithms.

Findings

Provides provably valid confidence measures for predictions

Enables control of prediction error rates

Applicable to a wide range of machine learning methods

Abstract

Recent advances in machine learning make it possible to design efficient prediction algorithms for data sets with huge numbers of parameters. This paper describes a new technique for "hedging" the predictions output by many such algorithms, including support vector machines, kernel ridge regression, kernel nearest neighbours, and by many other state-of-the-art methods. The hedged predictions for the labels of new objects include quantitative measures of their own accuracy and reliability. These measures are provably valid under the assumption of randomness, traditional in machine learning: the objects and their labels are assumed to be generated independently from the same probability distribution. In particular, it becomes possible to control (up to statistical fluctuations) the number of erroneous predictions by selecting a suitable confidence level. Validity being achieved automatically, the remaining goal of hedged prediction is efficiency: taking full account of the new objects' features and other available information to produce as accurate predictions as possible. This can be done successfully using the powerful machinery of modern machine learning.