Heaviside Set Constrained Optimization: Optimality and Newton Method

TL;DR

This paper directly addresses optimization problems involving the Heaviside step function, deriving optimality conditions and proposing a Newton method with quadratic convergence for improved binary status modeling.

Contribution

It introduces a novel approach to handle the Heaviside step function directly in optimization, including cone calculations, optimality conditions, and a new Newton method.

Findings

Derived tangent and normal cones for the feasible set

Established first-order optimality conditions

Developed a Newton method with quadratic convergence

Abstract

Data in the real world frequently involve binary status: truth or falsehood, positiveness or negativeness, similarity or dissimilarity, spam or non-spam, and to name a few, with applications into the regression, classification problems and so on. To characterize the binary status, one of the ideal functions is the Heaviside step function that returns one for one status and zero for the other. Hence, it is of dis-continuity. Because of this, the conventional approaches to deal with the binary status tremendously benefit from its continuous surrogates. In this paper, we target the Heaviside step function directly and study the Heaviside set constrained optimization: calculating the tangent and normal cones of the feasible set, establishing several first-order sufficient and necessary optimality conditions, as well as developing a Newton type method that enjoys locally quadratic convergence and excellent numerical performance.