Debiasing a First-order Heuristic for Approximate Bi-level Optimization

TL;DR

This paper analyzes the convergence issues of a popular heuristic for approximate bi-level optimization and introduces an unbiased variant with theoretical guarantees and practical adaptive schemes.

Contribution

It provides the first theoretical analysis of the bias in the FOM heuristic and proposes an unbiased FOM (UFOM) with convergence guarantees and adaptive strategies.

Findings

FOM can be biased and may not converge to stationary points.

UFOM is an unbiased alternative with constant memory complexity.

An adaptive UFOM scheme improves practical performance.

Abstract

Approximate bi-level optimization (ABLO) consists of (outer-level) optimization problems, involving numerical (inner-level) optimization loops. While ABLO has many applications across deep learning, it suffers from time and memory complexity proportional to the length $r$ of its inner optimization loop. To address this complexity, an earlier first-order method (FOM) was proposed as a heuristic that omits second derivative terms, yielding significant speed gains and requiring only constant memory. Despite FOM's popularity, there is a lack of theoretical understanding of its convergence properties. We contribute by theoretically characterizing FOM's gradient bias under mild assumptions. We further demonstrate a rich family of examples where FOM-based SGD does not converge to a stationary point of the ABLO objective. We address this concern by proposing an unbiased FOM (UFOM) enjoying constant memory complexity as a function of $r$. We characterize the introduced time-variance tradeoff, demonstrate convergence bounds, and find an optimal UFOM for a given ABLO problem. Finally, we propose an efficient adaptive UFOM scheme.