CPMLHO:Hyperparameter Tuning via Cutting Plane and Mixed-Level Optimization

TL;DR

This paper introduces CPMLHO, a novel hyperparameter optimization method combining cutting plane and mixed-level optimization to efficiently find superior hyperparameters with higher accuracy and faster convergence in neural networks.

Contribution

The paper proposes a new hyperparameter optimization approach using cutting plane constraints and mixed-level objectives, improving efficiency and accuracy over existing methods.

Findings

Automatically updates hyperparameters during training.

Achieves higher accuracy with faster convergence.

Outperforms existing hyperparameter tuning methods.

Abstract

The hyperparameter optimization of neural network can be expressed as a bilevel optimization problem. The bilevel optimization is used to automatically update the hyperparameter, and the gradient of the hyperparameter is the approximate gradient based on the best response function. Finding the best response function is very time consuming. In this paper we propose CPMLHO, a new hyperparameter optimization method using cutting plane method and mixed-level objective function.The cutting plane is added to the inner layer to constrain the space of the response function. To obtain more accurate hypergradient,the mixed-level can flexibly adjust the loss function by using the loss of the training set and the verification set. Compared to existing methods, the experimental results show that our method can automatically update the hyperparameters in the training process, and can find more superior hyperparameters with higher accuracy and faster convergence.