Best-first Search Algorithm for Non-convex Sparse Minimization

TL;DR

This paper introduces a best-first search algorithm for solving non-convex sparse minimization problems with convex objectives, providing a controllable trade-off between solution accuracy and computational effort.

Contribution

It develops an exact BFS-based method for NSM with general convex objectives, extending beyond quadratic cases and enabling solutions with bounded objective errors.

Findings

BFS effectively solves moderate-size NSM instances with non-quadratic objectives.

The method is faster than MIP-based approaches for quadratic objectives.

It offers a controllable error parameter to balance accuracy and computation.

Abstract

Non-convex sparse minimization (NSM), or $\ell_0$-constrained minimization of convex loss functions, is an important optimization problem that has many machine learning applications. NSM is generally NP-hard, and so to exactly solve NSM is almost impossible in polynomial time. As regards the case of quadratic objective functions, exact algorithms based on quadratic mixed-integer programming (MIP) have been studied, but no existing exact methods can handle more general objective functions including Huber and logistic losses; this is unfortunate since those functions are prevalent in practice. In this paper, we consider NSM with $\ell_2$-regularized convex objective functions and develop an algorithm by leveraging the efficiency of best-first search (BFS). Our BFS can compute solutions with objective errors at most $\Delta\ge0$, where $\Delta$ is a controllable hyper-parameter that balances the trade-off between the guarantee of objective errors and computation cost. Experiments demonstrate that our BFS is useful for solving moderate-size NSM instances with non-quadratic objectives and that BFS is also faster than the MIP-based method when applied to quadratic objectives.