Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks

TL;DR

This paper introduces Sparse Spectral Training (SST), a memory-efficient method for training Euclidean and hyperbolic neural networks that updates singular values and vectors selectively, outperforming existing low-rank approaches.

Contribution

SST is a novel spectral training approach that optimizes memory use by updating singular values and vectors selectively, improving over prior low-rank methods during pre-training.

Findings

SST reduces the perplexity gap by 97.4% on LLaMA-1.3B.

SST trains with only 18.7% of parameters of full-rank training.

SST outperforms existing memory reduction methods in various tasks.

Abstract

The growing demands on GPU memory posed by the increasing number of neural network parameters call for training approaches that are more memory-efficient. Previous memory reduction training techniques, such as Low-Rank Adaptation (LoRA) and ReLoRA, face challenges, with LoRA being constrained by its low-rank structure, particularly during intensive tasks like pre-training, and ReLoRA suffering from saddle point issues. In this paper, we propose Sparse Spectral Training (SST) to optimize memory usage for pre-training. SST updates all singular values and selectively updates singular vectors through a multinomial sampling method weighted by the magnitude of the singular values. Furthermore, SST employs singular value decomposition to initialize and periodically reinitialize low-rank parameters, reducing distortion relative to full-rank training compared to other low-rank methods. Through comprehensive testing on both Euclidean and hyperbolic neural networks across various tasks, SST demonstrates its ability to outperform existing memory reduction training methods and is comparable to full-rank training in various cases. On LLaMA-1.3B, with only 18.7\% of the parameters trainable compared to full-rank training (using a rank equivalent to 6\% of the embedding dimension), SST reduces the perplexity gap between other low-rank methods and full-rank training by 97.4\%. This result highlights SST as an effective parameter-efficient technique for model pre-training.