Maximin Optimization for Binary Regression

TL;DR

This paper investigates maximin optimization methods for binary regression problems, demonstrating their theoretical optimality in certain settings and practical effectiveness in neural networks and robust regression.

Contribution

It provides new theoretical insights into maximin techniques for binary regression and shows their practical success in various models.

Findings

Optimal in linear regression with low noise

Effective in robust regression with few outliers

Performs well in neural networks with non-convex loss

Abstract

We consider regression problems with binary weights. Such optimization problems are ubiquitous in quantized learning models and digital communication systems. A natural approach is to optimize the corresponding Lagrangian using variants of the gradient ascent-descent method. Such maximin techniques are still poorly understood even in the concave-convex case. The non-convex binary constraints may lead to spurious local minima. Interestingly, we prove that this approach is optimal in linear regression with low noise conditions as well as robust regression with a small number of outliers. Practically, the method also performs well in regression with cross entropy loss, as well as non-convex multi-layer neural networks. Taken together our approach highlights the potential of saddle-point optimization for learning constrained models.