Nonsmoothness in Machine Learning: specific structure, proximal identification, and applications

TL;DR

This paper explores the unique structures of nonsmooth optimization problems in machine learning, demonstrating how to leverage these structures for practical benefits like compression, acceleration, and dimension reduction.

Contribution

It characterizes the specific structure of nonsmooth problems in machine learning and provides practical methods to exploit this for improved optimization techniques.

Findings

Identification of problem structures in nonsmooth optimization

Methods for leveraging structure in compression and acceleration

Accessible presentation with examples and general results

Abstract

Nonsmoothness is often a curse for optimization; but it is sometimes a blessing, in particular for applications in machine learning. In this paper, we present the specific structure of nonsmooth optimization problems appearing in machine learning and illustrate how to leverage this structure in practice, for compression, acceleration, or dimension reduction. We pay a special attention to the presentation to make it concise and easily accessible, with both simple examples and general results.