Indefinite Kernel Logistic Regression with Concave-inexact-convex Procedure

TL;DR

This paper introduces an indefinite kernel logistic regression framework that leverages a concave-inexact-convex procedure to handle non-positive definite kernels, with proven convergence and competitive experimental performance.

Contribution

It develops a novel indefinite kernel logistic regression model using a difference of convex functions and proposes efficient algorithms with convergence guarantees.

Findings

Outperforms standard positive-definite kernel logistic regression.

Effective in both deterministic and stochastic settings.

Accelerates convergence with the proposed CCICP algorithms.

Abstract

In kernel methods, the kernels are often required to be positive definite, which restricts the use of many indefinite kernels. To consider those non-positive definite kernels, in this paper, we aim to build an indefinite kernel learning framework for kernel logistic regression. The proposed indefinite kernel logistic regression (IKLR) model is analysed in the Reproducing Kernel Kre\u{\i}n Spaces (RKKS) and then becomes non-convex. Using the positive decomposition of a non-positive definite kernel, the derived IKLR model can be decomposed into the difference of two convex functions. Accordingly, a concave-convex procedure is introduced to solve the non-convex optimization problem. Since the concave-convex procedure has to solve a sub-problem in each iteration, we propose a concave-inexact-convex procedure (CCICP) algorithm with an inexact solving scheme to accelerate the solving process. Besides, we propose a stochastic variant of CCICP to efficiently obtain a proximal solution, which achieves the similar purpose with the inexact solving scheme in CCICP. The convergence analyses of the above two variants of concave-convex procedure are conducted. By doing so, our method works effectively not only under a deterministic setting but also under a stochastic setting. Experimental results on several benchmarks suggest that the proposed IKLR model performs favorably against the standard (positive-definite) kernel logistic regression and other competitive indefinite learning based algorithms.