Improved Error Bounds Based on Worst Likely Assignments

TL;DR

This paper introduces a new statistic for permutation tests in worst likely assignment error bounds, enhancing their accuracy especially for classifiers with high precision on small datasets.

Contribution

It proposes an improved statistic for permutation tests that yields tighter error bounds for accurate classifiers on small datasets.

Findings

Enhanced error bounds for classifiers with small training sets

Effective permutation test statistic for worst likely assignments

Improved bounds for classifiers with high accuracy

Abstract

Error bounds based on worst likely assignments use permutation tests to validate classifiers. Worst likely assignments can produce effective bounds even for data sets with 100 or fewer training examples. This paper introduces a statistic for use in the permutation tests of worst likely assignments that improves error bounds, especially for accurate classifiers, which are typically the classifiers of interest.