Variance Reduced Random Relaxed Projection Method for Constrained Finite-sum Minimization Problems

TL;DR

This paper introduces a variance reduced random projection method for efficiently solving large-scale constrained convex optimization problems, improving convergence speed and computational efficiency over existing methods.

Contribution

The paper proposes a novel RPM that uses half-space projections and variance reduction, with theoretical convergence guarantees and practical advantages demonstrated through experiments.

Findings

Faster convergence rate in optimality gap with Lipschitz continuous gradients.

Reduced computational cost due to closed-form half-space projections.

Superior performance demonstrated on beamforming and classification problems.

Abstract

For many applications in signal processing and machine learning, we are tasked with minimizing a large sum of convex functions subject to a large number of convex constraints. In this paper, we devise a new random projection method (RPM) to efficiently solve this problem. Compared with existing RPMs, our proposed algorithm features two useful algorithmic ideas. First, at each iteration, instead of projecting onto the subset defined by one of the constraints, our algorithm only requires projecting onto a half-space approximation of the subset, which significantly reduces the computational cost as it admits a closed-form formula. Second, to exploit the structure that the objective is a sum, variance reduction is incorporated into our algorithm to further improve the performance. As theoretical contributions, under a novel error bound condition and other standard assumptions, we prove that the proposed RPM converges to an optimal solution and that both optimality and feasibility gaps vanish at a sublinear rate. In particular, via a new analysis framework, we show that our RPM attains a faster convergence rate in optimality gap than existing RPMs when the objective function has a Lipschitz continuous gradient, capitalizing the benefit of the variance reduction. We also provide sufficient conditions for the error bound condition to hold. Experiments on a beamforming problem and a robust classification problem are also presented to demonstrate the superiority of our RPM over existing ones.