Distribution free M-estimation

TL;DR

This paper characterizes when convex M-estimation problems can be solved without assumptions on data distribution, revealing new insights into distribution-free solvability and challenging classical learnability conditions.

Contribution

It provides a precise characterization of distribution-free solvability for convex M-estimation, identifying conditions that differ from traditional learnability criteria.

Findings

Lipschitz continuity of the loss is not necessary for distribution-free minimization

A clear dividing line between solvable and unsolvable problems is established

The conditions differ from classical learnability characterizations

Abstract

The basic question of delineating those statistical problems that are solvable without making any assumptions on the underlying data distribution has long animated statistics and learning theory. This paper characterizes when a convex M-estimation or stochastic optimization problem is solvable in such an assumption-free setting, providing a precise dividing line between solvable and unsolvable problems. The conditions we identify show, perhaps surprisingly, that Lipschitz continuity of the loss being minimized is not necessary for distribution free minimization, and they are also distinct from classical characterizations of learnability in machine learning.