Minimizing the Maximal Loss: How and Why?

TL;DR

This paper introduces an algorithm that transforms online learning methods to minimize the maximal loss, addressing robustness and generalization issues associated with traditional average loss minimization.

Contribution

It presents a novel algorithm for converting online algorithms into maximal loss minimizers and proposes robust variants to handle outliers.

Findings

The algorithm effectively minimizes maximal loss in various settings.

Better training accuracy can lead to improved generalization performance.

Robust versions handle outliers effectively.

Abstract

A commonly used learning rule is to approximately minimize the \emph{average} loss over the training set. Other learning algorithms, such as AdaBoost and hard-SVM, aim at minimizing the \emph{maximal} loss over the training set. The average loss is more popular, particularly in deep learning, due to three main reasons. First, it can be conveniently minimized using online algorithms, that process few examples at each iteration. Second, it is often argued that there is no sense to minimize the loss on the training set too much, as it will not be reflected in the generalization loss. Last, the maximal loss is not robust to outliers. In this paper we describe and analyze an algorithm that can convert any online algorithm to a minimizer of the maximal loss. We prove that in some situations better accuracy on the training set is crucial to obtain good performance on unseen examples. Last, we propose robust versions of the approach that can handle outliers.