On the structure of regularization paths for piecewise differentiable regularization terms

TL;DR

This paper investigates the structure of regularization paths for problems with piecewise differentiable regularization terms, showing they are piecewise smooth and classifying nonsmooth features, with applications in SVMs and penalty methods.

Contribution

It provides a theoretical framework for understanding the structure of regularization paths with piecewise differentiable penalties, extending previous results to more general cases.

Findings

Regularization paths are piecewise smooth.

Nonsmooth features can be classified.

Applications demonstrate the theory's practical relevance.

Abstract

Regularization is used in many different areas of optimization when solutions are sought which not only minimize a given function, but also possess a certain degree of regularity. Popular applications are image denoising, sparse regression and machine learning. Since the choice of the regularization parameter is crucial but often difficult, path-following methods are used to approximate the entire regularization path, i.e., the set of all possible solutions for all regularization parameters. Due to their nature, the development of these methods requires structural results about the regularization path. The goal of this article is to derive these results for the case of a smooth objective function which is penalized by a piecewise differentiable regularization term. We do this by treating regularization as a multiobjective optimization problem. Our results suggest that even in this general case, the regularization path is piecewise smooth. Moreover, our theory allows for a classification of the nonsmooth features that occur in between smooth parts. This is demonstrated in two applications, namely support-vector machines and exact penalty methods.