A multiobjective continuation method to compute the regularization path of deep neural networks

TL;DR

This paper introduces an efficient algorithm to compute the regularization path for high-dimensional deep neural networks by approximating the Pareto front in a multiobjective optimization setting, enhancing model interpretability and robustness.

Contribution

It presents the first scalable algorithm for approximating the regularization path in non-convex multiobjective problems with millions of parameters in deep neural networks.

Findings

Efficient approximation of the Pareto front for high-dimensional DNNs.

Knowledge of the regularization path improves network generalization.

Algorithm works with both deterministic and stochastic gradients.

Abstract

Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability of models (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a \emph{regularization path} connecting the sparsest solution in terms of the $\ell^1$ norm, i.e., zero weights and the non-regularized solution. Very recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity ($\ell^1$ norm) as two conflicting criteria and solving the resulting multiobjective optimization problem for low-dimensional DNN. However, due to the non-smoothness of the $\ell^1$ norm and the high number of parameters, this approach is not very efficient from a computational perspective for high-dimensional DNNs. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner for high-dimensional DNNs with millions of parameters. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization. To the best of our knowledge, this is the first algorithm to compute the regularization path for non-convex multiobjective optimization problems (MOPs) with millions of degrees of freedom.