On the relations of stochastic convex optimization problems with empirical risk minimization problems on $p$-norm balls

TL;DR

This paper investigates the relationship between stochastic convex optimization and empirical risk minimization on p-norm balls, analyzing how the choice of p influences sample complexity in machine learning and statistical problems.

Contribution

It extends existing results from Euclidean spaces to arbitrary p-norm balls, providing new insights into sample size requirements based on the parameter p.

Findings

Derived bounds on sample size for empirical risk minimization on p-norm balls

Analyzed the impact of p on the accuracy of empirical solutions

Extended theoretical results to a broader class of normed spaces

Abstract

In this paper, we consider convex stochastic optimization problems arising in machine learning applications (e.g., risk minimization) and mathematical statistics (e.g., maximum likelihood estimation). There are two main approaches to solve such kinds of problems, namely the Stochastic Approximation approach (online approach) and the Sample Average Approximation approach, also known as the Monte Carlo approach, (offline approach). In the offline approach, the problem is replaced by its empirical counterpart (the empirical risk minimization problem). The natural question is how to define the problem sample size, i.e., how many realizations should be sampled so that the quite accurate solution of the empirical problem be the solution of the original problem with the desired precision. This issue is one of the main issues in modern machine learning and optimization. In the last decade, a lot of significant advances were made in these areas to solve convex stochastic optimization problems on the Euclidean balls (or the whole space). In this work, we are based on these advances and study the case of arbitrary balls in the $\ell_p$-norms. We also explore the question of how the parameter $p$ affects the estimates of the required number of terms as a function of empirical risk.