A Mathematical Foundation for Robust Machine Learning based on Bias-Variance Trade-off

TL;DR

This paper develops a mathematical framework for robust machine learning by analyzing the impact of unequal sample contributions through bias-variance trade-off, enabling better comparison and improvement of algorithms.

Contribution

It introduces a unified theoretical foundation for robust machine learning, clarifies differences among strategies, and proposes a combined approach based on classical methods.

Findings

Provides definitions and properties for RML based on bias-variance trade-off

Analyzes and compares classical learning algorithms within the new framework

Proposes a unified method combining two classical strategies

Abstract

A common assumption in machine learning is that samples are independently and identically distributed (i.i.d). However, the contributions of different samples are not identical in training. Some samples are difficult to learn and some samples are noisy. The unequal contributions of samples has a considerable effect on training performances. Studies focusing on unequal sample contributions (e.g., easy, hard, noisy) in learning usually refer to these contributions as robust machine learning (RML). Weighing and regularization are two common techniques in RML. Numerous learning algorithms have been proposed but the strategies for dealing with easy/hard/noisy samples differ or even contradict with different learning algorithms. For example, some strategies take the hard samples first, whereas some strategies take easy first. Conducting a clear comparison for existing RML algorithms in dealing with different samples is difficult due to lack of a unified theoretical framework for RML. This study attempts to construct a mathematical foundation for RML based on the bias-variance trade-off theory. A series of definitions and properties are presented and proved. Several classical learning algorithms are also explained and compared. Improvements of existing methods are obtained based on the comparison. A unified method that combines two classical learning strategies is proposed.