# Estimating the standard error of cross-Validation-Based estimators of classifier performance

## TL;DR

This paper analyzes the variance of cross-validation estimators for classifier performance, proposes an influence function-based estimator, and compares its effectiveness to ad-hoc methods through experiments.

## Contribution

It introduces a novel influence function-based estimator for the variance of CV-based classifiers, addressing a gap in rigorous variance estimation methods.

## Key findings

- IF estimator has small RMS error but some bias
- Ad-hoc methods perform better despite lack of theoretical foundation
- Further research needed on smoothness and bias reduction

## Abstract

First, we analyze the variance of the Cross Validation (CV)-based estimators used for estimating the performance of classification rules. Second, we propose a novel estimator to estimate this variance using the Influence Function (IF) approach that had been used previously very successfully to estimate the variance of the bootstrap-based estimators. The motivation for this research is that, as the best of our knowledge, the literature lacks a rigorous method for estimating the variance of the CV-based estimators. What is available is a set of ad-hoc procedures that have no mathematical foundation since they ignore the covariance structure among dependent random variables. The conducted experiments show that the IF proposed method has small RMS error with some bias. However, surprisingly, the ad-hoc methods still work better than the IF-based method. Unfortunately, this is due to the lack of enough smoothness if compared to the bootstrap estimator. This opens the research for three points: (1) more comprehensive simulation study to clarify when the IF method win or loose; (2) more mathematical analysis to figure out why the ad-hoc methods work well; and (3) more mathematical treatment to figure out the connection between the appropriate amount of "smoothness" and decreasing the bias of the IF method.