Efficient Hyperparameter Tuning with Dynamic Accuracy Derivative-Free Optimization

TL;DR

This paper introduces a dynamic accuracy derivative-free optimization method for hyperparameter tuning that handles inexact evaluations, demonstrating improved robustness and efficiency over fixed accuracy methods in logistic classifier learning.

Contribution

The paper applies a novel dynamic accuracy derivative-free optimization approach to hyperparameter tuning, providing convergence guarantees and practical efficiency improvements.

Findings

The method is robust and efficient in hyperparameter tuning.

It outperforms fixed accuracy approaches in experiments.

Convergence guarantees are maintained despite inexact evaluations.

Abstract

Many machine learning solutions are framed as optimization problems which rely on good hyperparameters. Algorithms for tuning these hyperparameters usually assume access to exact solutions to the underlying learning problem, which is typically not practical. Here, we apply a recent dynamic accuracy derivative-free optimization method to hyperparameter tuning, which allows inexact evaluations of the learning problem while retaining convergence guarantees. We test the method on the problem of learning elastic net weights for a logistic classifier, and demonstrate its robustness and efficiency compared to a fixed accuracy approach. This demonstrates a promising approach for hyperparameter tuning, with both convergence guarantees and practical performance.