Problem-Parameter-Free Decentralized Bilevel Optimization

TL;DR

This paper introduces AdaSDBO, a fully problem-parameter-free decentralized bilevel optimization algorithm that adaptively adjusts stepsizes, eliminating the need for hyperparameter tuning and achieving competitive convergence rates.

Contribution

The paper presents AdaSDBO, a novel adaptive, single-loop decentralized bilevel optimization method that removes the reliance on problem-specific parameters for stepsize selection.

Findings

Achieves a convergence rate of (rac{1}{T})

Demonstrates robustness across various stepsize configurations

Performs competitively with state-of-the-art methods

Abstract

Decentralized bilevel optimization has garnered significant attention due to its critical role in solving large-scale machine learning problems. However, existing methods often rely on prior knowledge of problem parameters-such as smoothness, convexity, or communication network topologies-to determine appropriate stepsizes. In practice, these problem parameters are typically unavailable, leading to substantial manual effort for hyperparameter tuning. In this paper, we propose AdaSDBO, a fully problem-parameter-free algorithm for decentralized bilevel optimization with a single-loop structure. AdaSDBO leverages adaptive stepsizes based on cumulative gradient norms to update all variables simultaneously, dynamically adjusting its progress and eliminating the need for problem-specific hyperparameter tuning. Through rigorous theoretical analysis, we establish that AdaSDBO achieves a convergence rate of $\widetilde{\mathcal{O}}\left(\frac{1}{T}\right)$, matching the performance of well-tuned state-of-the-art methods up to polylogarithmic factors. Extensive numerical experiments demonstrate that AdaSDBO delivers competitive performance compared to existing decentralized bilevel optimization methods while exhibiting remarkable robustness across diverse stepsize configurations.