Teleportation With Null Space Gradient Projection for Optimization Acceleration

TL;DR

This paper introduces a null space gradient projection method for teleportation in optimization, enhancing convergence speed and computational efficiency across various neural network architectures.

Contribution

The paper presents a novel teleportation algorithm that projects gradients onto the null space, enabling efficient optimization for complex models like CNNs and Transformers.

Findings

Effective across multiple neural network architectures

Reduces computational cost of teleportation methods

Demonstrates broad applicability on benchmark datasets

Abstract

Optimization techniques have become increasingly critical due to the ever-growing model complexity and data scale. In particular, teleportation has emerged as a promising approach, which accelerates convergence of gradient descent-based methods by navigating within the loss invariant level set to identify parameters with advantageous geometric properties. Existing teleportation algorithms have primarily demonstrated their effectiveness in optimizing Multi-Layer Perceptrons (MLPs), but their extension to more advanced architectures, such as Convolutional Neural Networks (CNNs) and Transformers, remains challenging. Moreover, they often impose significant computational demands, limiting their applicability to complex architectures. To this end, we introduce an algorithm that projects the gradient of the teleportation objective function onto the input null space, effectively preserving the teleportation within the loss invariant level set and reducing computational cost. Our approach is readily generalizable from MLPs to CNNs, transformers, and potentially other advanced architectures. We validate the effectiveness of our algorithm across various benchmark datasets and optimizers, demonstrating its broad applicability.