A Fast Smoothing Newton Method for Bilevel Hyperparameter Optimization for SVC with Logistic Loss

TL;DR

This paper introduces a fast smoothing Newton method for solving a bilevel hyperparameter optimization problem in SVC with logistic loss, achieving efficient and accurate solutions with superlinear convergence.

Contribution

The paper reformulates hyperparameter tuning for SVC with logistic loss as a single-level NLP and applies a smoothing Newton method, demonstrating superlinear convergence and improved efficiency.

Findings

The proposed method converges superlinearly.

It achieves competitive results with less computational time.

Strict local minimizers are verified both numerically and theoretically.

Abstract

Support vector classification (SVC) with logistic loss has excellent theoretical properties in classification problems where the label values are not continuous. In this paper, we reformulate the hyperparameter selection for SVC with logistic loss as a bilevel optimization problem in which the upper-level problem and the lower-level problem are both based on logistic loss. The resulting bilevel optimization model is converted to a single-level nonlinear programming (NLP) problem based on the KKT conditions of the lower-level problem. Such NLP contains a set of nonlinear equality constraints and a simple lower bound constraint. The second-order sufficient condition is characterized, which guarantees that the strict local optimizers are obtained. To solve such NLP, we apply the smoothing Newton method proposed in \cite{Liang} to solve the KKT conditions, which contain one pair of complementarity constraints. We show that the smoothing Newton method has a superlinear convergence rate. Extensive numerical results verify the efficiency of the proposed approach and strict local minimizers can be achieved both numerically and theoretically. In particular, compared with other methods, our algorithm can achieve competitive results while consuming less time than other methods.