Cogradient Descent for Dependable Learning

TL;DR

This paper introduces Cogradient Descent (CoGD), a novel optimization algorithm that better handles coupled variables in bilinear models, improving training and performance in tasks like image reconstruction and CNN training.

Contribution

The paper proposes CoGD, a new gradient-based method that systematically coordinates gradients of coupled variables, especially when sparsity constraints are involved, enhancing bilinear model optimization.

Findings

CoGD outperforms existing methods in image reconstruction and inpainting.

CoGD improves CNN training and model capacity.

Extensive experiments demonstrate significant performance gains.

Abstract

Conventional gradient descent methods compute the gradients for multiple variables through the partial derivative. Treating the coupled variables independently while ignoring the interaction, however, leads to an insufficient optimization for bilinear models. In this paper, we propose a dependable learning based on Cogradient Descent (CoGD) algorithm to address the bilinear optimization problem, providing a systematic way to coordinate the gradients of coupling variables based on a kernelized projection function. CoGD is introduced to solve bilinear problems when one variable is with sparsity constraint, as often occurs in modern learning paradigms. CoGD can also be used to decompose the association of features and weights, which further generalizes our method to better train convolutional neural networks (CNNs) and improve the model capacity. CoGD is applied in representative bilinear problems, including image reconstruction, image inpainting, network pruning and CNN training. Extensive experiments show that CoGD improves the state-of-the-arts by significant margins. Code is available at {https://github.com/bczhangbczhang/CoGD}.