Robust Learning Meets Quasar-Convex Optimization: Inexact High-Order Proximal-Point Methods

TL;DR

This paper introduces HiPPA, an inexact high-order proximal-point method for robust learning formulated as quasar-convex optimization, demonstrating global convergence and strong empirical performance.

Contribution

It formulates robust learning problems as quasar-convex optimization and proposes HiPPA, a novel method with proven convergence properties and empirical success.

Findings

HiPPA achieves global convergence to minima.

It attains linear or superlinear convergence rates.

Numerical experiments show strong empirical performance.

Abstract

Robust learning aims to maintain model performance under noise, corruption, and distributional shifts, which are prevalent in modern machine learning applications. This work shows that examples of robust learning problems can be formulated as (strongly) quasar-convex optimization problems, which admit a benign landscape with no saddle points. We then propose HiPPA, an inexact high-order proximal-point method that employs a model-value gap to control the inexactness of subproblem solutions. Notably, we prove global convergence of HiPPA to global minima and establish that it attains a (local) linear or superlinear convergence rate, depending on the regularization order and inexactness control. Our numerical experiments on robust feature-alignment distillation indicate strong empirical performance of HiPPA and results consistent with our theoretical findings.