Provably Data-driven Multiple Hyper-parameter Tuning with Structured Loss Function

TL;DR

This paper develops a new theoretical framework for data-driven hyperparameter tuning in multi-dimensional settings, providing generalization guarantees and lower bounds, applicable to structured models like weighted group lasso.

Contribution

It introduces the first comprehensive framework for multi-dimensional hyperparameter tuning guarantees, leveraging real algebraic geometry for sharper bounds.

Findings

Established the first generalization guarantees for multi-dimensional hyperparameter tuning.

Derived lower bounds for the hyperparameter tuning problem.

Applied the framework to models like weighted group lasso and fused lasso.

Abstract

Data-driven algorithm design automates hyperparameter tuning, but its statistical foundations remain limited because model performance can depend on hyperparameters in implicit and highly non-smooth ways. Existing guarantees focus on the simple case of a one-dimensional (scalar) hyperparameter. This leaves the practically important, multi-dimensional hyperparameter tuning setting unresolved. We address this open question by establishing the first general framework for establishing generalization guarantees for tuning multi-dimensional hyperparameters in data-driven settings. Our approach strengthens the generalization guarantee framework for semi-algebraic function classes by exploiting tools from real algebraic geometry, yielding sharper, more broadly applicable guarantees. For completeness, we also instantiate the first lower bound for this general setting. We further extend the analysis to hyperparameter tuning using the validation loss under minimal assumptions, and derive improved bounds when additional structure is available. Finally, we demonstrate the scope of the framework with new learnability results, including data-driven weighted group lasso and weighted fused lasso.