Mixing Deep Learning and Multiple Criteria Optimization: An Application to Distributed Learning with Multiple Datasets

TL;DR

This paper explores a novel approach combining deep learning with multiple criteria optimization for distributed learning across multiple datasets, providing stability analysis and practical implementation with digit classification experiments.

Contribution

It introduces a scalarization method for multi-criteria optimization in deep learning and extends it to multiple datasets, enhancing model robustness and reducing bias.

Findings

Stability of efficient solutions under data perturbations

Effective scalarization approach for multi-criteria learning

Successful digit classification experiments with MNIST

Abstract

The training phase is the most important stage during the machine learning process. In the case of labeled data and supervised learning, machine training consists in minimizing the loss function subject to different constraints. In an abstract setting, it can be formulated as a multiple criteria optimization model in which each criterion measures the distance between the output associated with a specific input and its label. Therefore, the fitting term is a vector function and its minimization is intended in the Pareto sense. We provide stability results of the efficient solutions with respect to perturbations of input and output data. We then extend the same approach to the case of learning with multiple datasets. The multiple dataset environment is relevant when reducing the bias due to the choice of a specific training set. We propose a scalarization approach to implement this model and numerical experiments in digit classification using MNIST data.