Quasi-Self-Concordant Optimization with Lewis Weights

TL;DR

This paper introduces a trust-region based optimization algorithm for quasi-self-concordant functions that improves computational efficiency by reducing the number of linear system solves, leveraging Lewis weights and IRLS techniques.

Contribution

It presents a novel trust-region algorithm with an oracle that uses fewer linear system solves, improving over previous methods for quasi-self-concordant optimization.

Findings

Reduces oracle complexity to rac{d^{1/3}}{ ext{epsilon}}

Achieves rac{(d^{1/3}/ ext{epsilon}+1/ ext{epsilon}^2) ext{log}(n/ ext{epsilon})} linear system solves for rac{1+ ext{epsilon}}{approximate} solutions

Demonstrates significant runtime improvements over standard solvers

Abstract

In this paper, we study the problem $\min_{x\in \mathbb{R}^{d},Nx=v}\sum_{i=1}^{n}f((Ax-b)_{i})$ for a quasi-self-concordant function $f:\mathbb{R}\to\mathbb{R}$, where $A,N$ are $n\times d$ and $m\times d$ matrices, $b,v$ are vectors of length $n$ and $m$ with $n\ge d.$ We show an algorithm based on a trust-region method with an oracle that can be implemented using $\widetilde{O}(d^{1/3})$ linear system solves, improving the $\widetilde{O}(n^{1/3})$ oracle by {[}Adil-Bullins-Sachdeva, NeurIPS 2021{]}. Our implementation of the oracle relies on solving the overdetermined $\ell_{\infty}$-regression problem $\min_{x\in\mathbb{R}^{d},Nx=v}\|Ax-b\|_{\infty}$. We provide an algorithm that finds a $(1+\epsilon)$-approximate solution to this problem using $O((d^{1/3}/\epsilon+1/\epsilon^{2})\log(n/\epsilon))$ linear system solves. This algorithm leverages $\ell_{\infty}$ Lewis weight overestimates and achieves this iteration complexity via a simple lightweight IRLS approach, inspired by the work of {[}Ene-Vladu, ICML 2019{]}. Experimentally, we demonstrate that our algorithm significantly improves the runtime of the standard CVX solver.