Solving Stackelberg Prediction Game with Least Squares Loss via Spherically Constrained Least Squares Reformulation

TL;DR

This paper introduces a novel reformulation of the Stackelberg prediction game with least squares loss as a spherically constrained least squares problem, enabling faster solutions for large-scale datasets.

Contribution

It proposes a new nonlinear reformulation of SPG-LS as SCLS and demonstrates efficient algorithms suitable for large-scale problems.

Findings

Achieves $ ilde{O}(N/\sqrt{\epsilon})$ complexity for approximate solutions.

Two factorization-free algorithms are effective for large datasets.

Significantly faster solutions compared to existing methods.

Abstract

The Stackelberg prediction game (SPG) is popular in characterizing strategic interactions between a learner and an attacker. As an important special case, the SPG with least squares loss (SPG-LS) has recently received much research attention. Although initially formulated as a difficult bi-level optimization problem, SPG-LS admits tractable reformulations which can be polynomially globally solved by semidefinite programming or second order cone programming. However, all the available approaches are not well-suited for handling large-scale datasets, especially those with huge numbers of features. In this paper, we explore an alternative reformulation of the SPG-LS. By a novel nonlinear change of variables, we rewrite the SPG-LS as a spherically constrained least squares (SCLS) problem. Theoretically, we show that an $\epsilon$ optimal solution to the SCLS (and the SPG-LS) can be achieved in $\tilde{O}(N/\sqrt{\epsilon})$ floating-point operations, where $N$ is the number of nonzero entries in the data matrix. Practically, we apply two well-known methods for solving this new reformulation, i.e., the Krylov subspace method and the Riemannian trust region method. Both algorithms are factorization free so that they are suitable for solving large scale problems. Numerical results on both synthetic and real-world datasets indicate that the SPG-LS, equipped with the SCLS reformulation, can be solved orders of magnitude faster than the state of the art.