A Concentration of Measure Framework to study convex problems and other implicit formulation problems in machine learning

TL;DR

This paper introduces a concentration of measure framework to analyze solutions of convex optimization problems in machine learning, enabling precise characterization of their behavior and performance.

Contribution

It develops a novel framework linking measure concentration to implicit convex problems, facilitating analysis of algorithms like logistic regression and lasso.

Findings

Provides bounds on the moments of solutions in convex problems

Characterizes the generalization error of classifiers

Applies to a wide range of implicit machine learning models

Abstract

This paper provides a framework to show the concentration of solutions $Y^*$ to convex minimizing problem where the objective function $\phi(X)(Y)$ depends on some random vector $X$ satisfying concentration of measure hypotheses. More precisely, the convex problem translates into a contractive fixed point equation that ensure the transmission of the concentration from $X$ to $Y^*$. This result is of central interest to characterize many machine learning algorithms which are defined through implicit equations (e.g., logistic regression, lasso, boosting, etc.). Based on our framework, we provide precise estimations for the first moments of the solution $Y^*$, when $X= (x_1,\ldots, x_n)$ is a data matrix of independent columns and $\phi(X)(y)$ writes as a sum $\frac{1}{n}\sum_{i=1}^n h_i(x_i^TY)$. That allows to describe the behavior and performance (e.g., generalization error) of a wide variety of machine learning classifiers.