Empirical Hypothesis Space Reduction
Akihiro Yabe, Takanori Maehara

TL;DR
This paper introduces an empirical hypothesis space reduction algorithm that accelerates generalization error convergence in regularized empirical risk minimization, especially in high-dimensional settings, by reducing the hypothesis space without losing the true optimum.
Contribution
It proposes a novel algorithm for empirical hypothesis space reduction that achieves faster, dimension-independent convergence of generalization error.
Findings
Achieves $O(rac{ ext{log} n}{n})$ convergence rate independent of hypothesis space dimension.
Reduces hypothesis space with high probability without losing the true optimum.
Accelerates generalization error convergence in high-dimensional regularized learning.
Abstract
Selecting appropriate regularization coefficients is critical to performance with respect to regularized empirical risk minimization problems. Existing theoretical approaches attempt to determine the coefficients in order for regularized empirical objectives to be upper-bounds of true objectives, uniformly over a hypothesis space. Such an approach is, however, known to be over-conservative, especially in high-dimensional settings with large hypothesis space. In fact, an existing generalization error bound in variance-based regularization is , where is the dimension of hypothesis space, and thus the number of samples required for convergence linearly increases with respect to . This paper proposes an algorithm that calculates regularization coefficient, one which results in faster convergence of generalization error and whose leading term…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Algorithms · Machine Learning and Data Classification · Optimization and Search Problems
