Statistical Robustness of Empirical Risks in Machine Learning

TL;DR

This paper investigates the robustness and stability of empirical risks in machine learning models within RKHS, especially under noisy data conditions, providing theoretical guarantees for estimator reliability and convergence.

Contribution

It introduces new conditions for stability of expected risk under data perturbations and analyzes the robustness of estimators using advanced metrics like Prokhorov and Kantorovich.

Findings

Derived conditions for stable expected risk under data noise.

Analyzed the robustness of estimators with respect to data contamination.

Identified metrics ensuring uniform asymptotic consistency of estimators.

Abstract

This paper studies convergence of empirical risks in reproducing kernel Hilbert spaces (RKHS). A conventional assumption in the existing research is that empirical training data do not contain any noise but this may not be satisfied in some practical circumstances. Consequently the existing convergence results do not provide a guarantee as to whether empirical risks based on empirical data are reliable or not when the data contain some noise. In this paper, we fill out the gap in a few steps. First, we derive moderate sufficient conditions under which the expected risk changes stably (continuously) against small perturbation of the probability distribution of the underlying random variables and demonstrate how the cost function and kernel affect the stability. Second, we examine the difference between laws of the statistical estimators of the expected optimal loss based on pure data and contaminated data using Prokhorov metric and Kantorovich metric and derive some qualitative and quantitative statistical robustness results. Third, we identify appropriate metrics under which the statistical estimators are uniformly asymptotically consistent. These results provide theoretical grounding for analysing asymptotic convergence and examining reliability of the statistical estimators in a number of well-known machine learning models.