Stochastic block coordinate and function alternation for multi-objective optimization and learning

TL;DR

This paper introduces a stochastic block coordinate and function alternation framework for multi-objective optimization, reducing computational costs and effectively exploring Pareto fronts in large-scale problems.

Contribution

It proposes a novel alternating optimization framework that improves efficiency and convergence guarantees for multi-objective problems across various stochastic settings.

Findings

Outperforms non-alternating methods in multi-target regression.

Achieves classical convergence rates in convex, non-convex, and Polyak-Lojasiewicz settings.

Effectively explores Pareto front with reduced computational cost.

Abstract

Multi-objective optimization is central to many engineering and machine learning applications, where multiple objectives must be optimized in balance. While multi-gradient based optimization methods combine these objectives in each step, such methods require computing gradients with respect to all variables at every iteration, resulting in high computational costs in large-scale settings. In this work, we propose a framework that simultaneously alternates the optimization of each objective and the (stochastic) gradient update with respect to each variable block. Our framework reduces per-iteration computational cost while enabling exploration of the Pareto front by allocating a prescribed number of gradient steps to each objective. We establish rigorous convergence guarantees across several stochastic smooth settings, including convex, non-convex, and Polyak-Lojasiewicz conditions, recovering classical convergence rates of single-objective methods. Numerical experiments demonstrate that our framework outperforms non-alternating methods on multi-target regression and produces a competitive Pareto front approximation, highlighting its computational efficiency and practical effectiveness.