Provably tuning the ElasticNet across instances
Maria-Florina Balcan, Mikhail Khodak, Dravyansh Sharma, Ameet, Talwalkar

TL;DR
This paper provides the first provable guarantees for tuning ElasticNet regularization parameters across multiple instances, including online and classification settings, using a novel structural analysis of the loss function.
Contribution
It introduces a structural characterization of ElasticNet loss as a piecewise-rational function, enabling generalization guarantees without strong data assumptions.
Findings
Bounded the complexity of ElasticNet loss functions.
Established generalization guarantees for tuning ElasticNet coefficients.
Achieved vanishing regret in online learning for ElasticNet tuning.
Abstract
An important unresolved challenge in the theory of regularization is to set the regularization coefficients of popular techniques like the ElasticNet with general provable guarantees. We consider the problem of tuning the regularization parameters of Ridge regression, LASSO, and the ElasticNet across multiple problem instances, a setting that encompasses both cross-validation and multi-task hyperparameter optimization. We obtain a novel structural result for the ElasticNet which characterizes the loss as a function of the tuning parameters as a piecewise-rational function with algebraic boundaries. We use this to bound the structural complexity of the regularized loss functions and show generalization guarantees for tuning the ElasticNet regression coefficients in the statistical setting. We also consider the more challenging online learning setting, where we show vanishing average…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsMachine Learning and Algorithms · Machine Learning and Data Classification · Advanced Bandit Algorithms Research
