Gradient directions and relative inexactness in optimization and machine learning

TL;DR

This paper explores how noise affects gradient directions in optimization, providing theoretical insights and experimental validation across various machine learning tasks.

Contribution

It introduces a theoretical framework for understanding gradient inexactness due to noise and demonstrates its implications through experiments.

Findings

Noise causes gradients to deviate with an acute angle from true directions

Theoretical models predict the impact of noise on optimization

Experimental results confirm the influence of gradient inexactness in ML tasks

Abstract

In this paper, we investigate the influence of noise giving an estimate of the gradient having a acute angle with the original. Noise amplitude has a relative model. The work offers both theoretical calculations and theorems, as well as experimental results. Classic machine learning problems were chosen as experiments -- linear and logistic regression, computer vision and natural language processing.