The Sample Complexity of Parameter-Free Stochastic Convex Optimization

TL;DR

This paper investigates the sample complexity of stochastic convex optimization without known problem parameters, introducing adaptive methods for tuning and regularization that improve performance and avoid overfitting.

Contribution

It proposes a reliable model selection technique and a regularization-based method for parameter-free stochastic convex optimization, enhancing adaptability and theoretical guarantees.

Findings

Model selection method matches optimal sample complexity up to log-log factors.

Regularization method adapts perfectly to unknown distance to optimality.

Experiments show improved performance in few-shot learning and shape counting tasks.

Abstract

We study the sample complexity of stochastic convex optimization when problem parameters, e.g., the distance to optimality, are unknown. We pursue two strategies. First, we develop a reliable model selection method that avoids overfitting the validation set. This method allows us to generically tune the learning rate of stochastic optimization methods to match the optimal known-parameter sample complexity up to $\log\log$ factors. Second, we develop a regularization-based method that is specialized to the case that only the distance to optimality is unknown. This method provides perfect adaptability to unknown distance to optimality, demonstrating a separation between the sample and computational complexity of parameter-free stochastic convex optimization. Combining these two methods allows us to simultaneously adapt to multiple problem structures. Experiments performing few-shot learning on CIFAR-10 by fine-tuning CLIP models and prompt engineering Gemini to count shapes indicate that our reliable model selection method can help mitigate overfitting to small validation sets.