Efficient Learning With Sine-Activated Low-rank Matrices

TL;DR

This paper introduces a sine-activated low-rank matrix decomposition method that improves the accuracy of parameter-efficient neural network models without sacrificing their compactness, applicable across various architectures.

Contribution

The authors propose a novel sine-activated low-rank decomposition framework that enhances model performance while maintaining parameter efficiency, serving as a plug-in for existing low-rank models.

Findings

Improved accuracy in Vision Transformers, LLMs, NeRF, and 3D shape modeling.

Preserves parameter efficiency while increasing effective rank.

Demonstrated versatility across multiple neural network architectures.

Abstract

Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates a sinusoidal function within the low-rank decomposition process. This approach not only preserves the benefits of the parameter efficiency characteristic of low-rank methods but also increases the decomposition's rank, thereby enhancing model performance. Our method proves to be a plug in enhancement for existing low-rank models, as evidenced by its successful application in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF) and 3D shape modelling.