CaAdam: Improving Adam optimizer using connection aware methods

TL;DR

CaAdam is a novel optimizer inspired by Adam that leverages architectural information like layer depth and connectivity to improve convergence speed and accuracy in neural network training.

Contribution

This paper introduces CaAdam, an architecture-aware optimizer that dynamically adjusts learning rates based on network structure, a feature absent in traditional optimizers.

Findings

Faster convergence on standard datasets

Higher accuracy compared to Adam

Effective use of structural proxies for optimization

Abstract

We introduce a new method inspired by Adam that enhances convergence speed and achieves better loss function minima. Traditional optimizers, including Adam, apply uniform or globally adjusted learning rates across neural networks without considering their architectural specifics. This architecture-agnostic approach is deeply embedded in most deep learning frameworks, where optimizers are implemented as standalone modules without direct access to the network's structural information. For instance, in popular frameworks like Keras or PyTorch, optimizers operate solely on gradients and parameters, without knowledge of layer connectivity or network topology. Our algorithm, CaAdam, explores this overlooked area by introducing connection-aware optimization through carefully designed proxies of architectural information. We propose multiple scaling methodologies that dynamically adjust learning rates based on easily accessible structural properties such as layer depth, connection counts, and gradient distributions. This approach enables more granular optimization while working within the constraints of current deep learning frameworks. Empirical evaluations on standard datasets (e.g., CIFAR-10, Fashion MNIST) show that our method consistently achieves faster convergence and higher accuracy compared to standard Adam optimizer, demonstrating the potential benefits of incorporating architectural awareness in optimization strategies.