Convexification of Learning from Constraints

TL;DR

This paper develops convex relaxations for learning problems with label constraints, transforming non-convex mixed integer programs into convex forms to enable more efficient optimization.

Contribution

It characterizes the tightest convex extension of the objective function and introduces efficiently computable looser convex relaxations for constrained learning problems.

Findings

Characterization of the tightest convex extension via Legendre-Fenchel biconjugate

Efficient computation of looser convex relaxations for common functions

Closed-form solutions for certain decompositions

Abstract

Regularized empirical risk minimization with constrained labels (in contrast to fixed labels) is a remarkably general abstraction of learning. For common loss and regularization functions, this optimization problem assumes the form of a mixed integer program (MIP) whose objective function is non-convex. In this form, the problem is resistant to standard optimization techniques. We construct MIPs with the same solutions whose objective functions are convex. Specifically, we characterize the tightest convex extension of the objective function, given by the Legendre-Fenchel biconjugate. Computing values of this tightest convex extension is NP-hard. However, by applying our characterization to every function in an additive decomposition of the objective function, we obtain a class of looser convex extensions that can be computed efficiently. For some decompositions, common loss and regularization functions, we derive a closed form.