# Concavifiability and convergence: necessary and sufficient conditions for gradient descent analysis

## TL;DR

This paper introduces the concept of concavifiability as a key property for analyzing gradient descent convergence, providing necessary and sufficient conditions that extend beyond traditional Lipschitz gradient assumptions.

## Contribution

It establishes concavifiability as a fundamental property for convergence analysis, derives the concavifier constant, and applies these insights to neural network training.

## Key findings

- Concavifiability is necessary and sufficient for gradient descent convergence.
- Any Lipschitz gradient function is concavifiable.
- Bounds on the concavifier enable fixed step size selection in neural networks.

## Abstract

Convergence of the gradient descent algorithm has been attracting renewed interest due to its utility in deep learning applications. Even as multiple variants of gradient descent were proposed, the assumption that the gradient of the objective is Lipschitz continuous remained an integral part of the analysis until recently. In this work, we look at convergence analysis by focusing on a property that we term as concavifiability, instead of Lipschitz continuity of gradients. We show that concavifiability is a necessary and sufficient condition to satisfy the upper quadratic approximation which is key in proving that the objective function decreases after every gradient descent update. We also show that any gradient Lipschitz function satisfies concavifiability. A constant known as the concavifier analogous to the gradient Lipschitz constant is derived which is indicative of the optimal step size. As an application, we demonstrate the utility of finding the concavifier the in convergence of gradient descent through an example inspired by neural networks. We derive bounds on the concavifier to obtain a fixed step size for a single hidden layer ReLU network.