Data-Dependent Complexity of First-Order Methods for Binary Classification

TL;DR

This paper analyzes the data-dependent convergence behavior of first-order methods like FISTA for binary classification problems, providing bounds and early stopping criteria that improve efficiency in large-scale data science tasks.

Contribution

It introduces data-dependent iteration bounds and early stopping criteria for FISTA applied to binary classification, enhancing convergence analysis and practical efficiency.

Findings

FISTA residual converges to the infimal displacement vector in ESP

Data-dependent iteration bounds scale as 1/δ_A^2

Early stopping yields well-classified hyperplanes with fewer iterations

Abstract

Large-scale problems in data science are often modeled with optimization, and the optimization model is usually solved with first-order methods that may converge at a sublinear rate. Therefore, it is of interest to terminate the optimization algorithm as soon as the underlying data science task is accomplished. We consider FISTA for solving two binary classification problems: the ellipsoid separation problem (ESP), and the soft-margin support-vector machine (SVM). For the ESP, we cast the dual second-order cone program into a form amenable to FISTA and show that the FISTA residual converges to the infimal displacement vector of the primal-dual hybrid gradient (PDHG) algorithm, that directly encodes a separating hyperplane. We further derive a data-dependent iteration upper bound scaling as $\mathcal{O}(1/\delta_{\mathcal{A}}^2)$, where $\delta_{\mathcal{A}}$ is the minimal perturbation that destroys separability. For the SVM, we propose a strongly-concave perturbed dual that admits efficient FISTA updates under a linear time projection scheme, and with our parameter choices, the objective has small condition number, enabling rapid convergence. We prove that, under a reasonable data model, early-stopped iterates identify well-classified points and yield a hyperplane that exactly separates them, where the accuracy required of the dual iterate is governed by geometric properties of the data. In particular, the proposed early-stopping criteria diminish the need for hard-to-select tolerance-based stopping conditions. Our numerical experiments on ESP instances derived from MNIST data and on soft-margin SVM benchmarks indicate competitive runtimes and substantial speedups from stopping early.