Computational Complexity of Sub-Linear Convergent Algorithms

TL;DR

This paper introduces a novel adaptive sampling approach that geometrically increases sample size to solve ERM problems efficiently with sublinear convergence, supported by theoretical proofs and experimental validation.

Contribution

It proposes a new adaptive sampling method that reduces computational complexity for first-order optimization algorithms with sublinear convergence.

Findings

Theoretical proof of the effectiveness of the adaptive sampling approach.

Experimental results show improved efficiency over standard gradient descent.

Adaptive sampling enhances convergence speed with lower computational costs.

Abstract

Optimizing machine learning algorithms that are used to solve the objective function has been of great interest. Several approaches to optimize common algorithms, such as gradient descent and stochastic gradient descent, were explored. One of these approaches is reducing the gradient variance through adaptive sampling to solve large-scale optimization's empirical risk minimization (ERM) problems. In this paper, we will explore how starting with a small sample and then geometrically increasing it and using the solution of the previous sample ERM to compute the new ERM. This will solve ERM problems with first-order optimization algorithms of sublinear convergence but with lower computational complexity. This paper starts with theoretical proof of the approach, followed by two experiments comparing the gradient descent with the adaptive sampling of the gradient descent and ADAM with adaptive sampling ADAM on different datasets.