A Multi-objective Newton Optimization Algorithm for Hyper-Parameter Search

TL;DR

This paper introduces a Newton-based multi-objective optimization algorithm for hyperparameter tuning in neural networks, demonstrating faster convergence and improved detection metrics over Bayesian methods.

Contribution

It presents a novel Newton-Raphson based approach with regularization for efficient multi-objective hyperparameter search in deep learning models.

Findings

Faster convergence to optimal parameters compared to Bayesian optimization.

Achieved higher true positive and lower false positive rates.

Parameter oscillation observed due to stochastic data effects.

Abstract

This study proposes a Newton based multiple objective optimization algorithm for hyperparameter search. The first order differential (gradient) is calculated using finite difference method and a gradient matrix with vectorization is formed for fast computation. The Newton Raphson iterative solution is used to update model parameters with iterations, and a regularization term is included to eliminate the singularity issue. The algorithm is applied to search the optimal probability threshold (a vector of eight parameters) for a multiclass object detection problem of a convolutional neural network. The algorithm quickly finds the improved parameter values to produce an overall higher true positive (TP) and lower false positive (FP) rates, as compared to using the default value of 0.5. In comparison, the Bayesian optimization generates lower performance in the testing case. However, the performance and parameter values may oscillate for some cases during iterations, which may be due to the data driven stochastic nature of the subject. Therefore, the optimal parameter value can be identified from a list of iteration steps according to the optimal TP and FP results.