Cogradient Descent for Bilinear Optimization

TL;DR

This paper introduces Cogradient Descent (CoGD), a novel optimization algorithm for bilinear models that synchronizes gradient updates of coupled variables, improving performance in tasks like image reconstruction and network pruning.

Contribution

The paper proposes a theoretical framework and algorithm for synchronized gradient descent in bilinear optimization, addressing issues of vanishing gradients and asynchronous updates.

Findings

Significant performance improvements over state-of-the-art methods.

Effective in applications like image reconstruction, inpainting, and network pruning.

Enhances optimization stability and convergence in bilinear models.

Abstract

Conventional learning methods simplify the bilinear model by regarding two intrinsically coupled factors independently, which degrades the optimization procedure. One reason lies in the insufficient training due to the asynchronous gradient descent, which results in vanishing gradients for the coupled variables. In this paper, we introduce a Cogradient Descent algorithm (CoGD) to address the bilinear problem, based on a theoretical framework to coordinate the gradient of hidden variables via a projection function. We solve one variable by considering its coupling relationship with the other, leading to a synchronous gradient descent to facilitate the optimization procedure. Our algorithm is applied to solve problems with one variable under the sparsity constraint, which is widely used in the learning paradigm. We validate our CoGD considering an extensive set of applications including image reconstruction, inpainting, and network pruning. Experiments show that it improves the state-of-the-art by a significant margin.