Single-Solution Hypervolume Maximization and its use for Improving Generalization of Neural Networks

TL;DR

This paper proposes a hypervolume maximization approach with a single solution as an alternative to mean loss minimization, offering theoretical insights and empirical validation that can improve neural network generalization.

Contribution

It introduces a hypervolume maximization method with a controllable hyperparameter, bridging the gap between max and mean loss minimization, and demonstrates its effectiveness on MNIST.

Findings

Hypervolume maximization can mimic mean loss minimization.

Adjusting the hyperparameter influences focus on higher-loss samples.

Achieved 20% reduction in test classification error.

Abstract

This paper introduces the hypervolume maximization with a single solution as an alternative to the mean loss minimization. The relationship between the two problems is proved through bounds on the cost function when an optimal solution to one of the problems is evaluated on the other, with a hyperparameter to control the similarity between the two problems. This same hyperparameter allows higher weight to be placed on samples with higher loss when computing the hypervolume's gradient, whose normalized version can range from the mean loss to the max loss. An experiment on MNIST with a neural network is used to validate the theory developed, showing that the hypervolume maximization can behave similarly to the mean loss minimization and can also provide better performance, resulting on a 20% reduction of the classification error on the test set.