MCNC: Manifold-Constrained Reparameterization for Neural Compression
Chayne Thrash, Ali Abbasi, Reed Andreas, Parsa Nooralinejad, Soroush, Abbasi Koohpayegani, Hamed Pirsiavash, Soheil Kolouri
TL;DR
This paper introduces MCNC, a novel neural network compression technique that constrains parameters to low-dimensional manifolds, achieving high compression rates while maintaining accuracy across diverse tasks.
Contribution
The paper proposes a new manifold-constrained reparameterization method for neural compression, outperforming existing techniques in efficiency and effectiveness.
Findings
Achieves unprecedented compression rates across tasks.
Outperforms state-of-the-art baselines in accuracy and reconstruction.
Effective across vision and NLP models.
Abstract
The outstanding performance of large foundational models across diverse tasks, from computer vision to speech and natural language processing, has significantly increased their demand. However, storing and transmitting these models poses significant challenges due to their massive size (e.g., 750GB for Llama 3.1 405B). Recent literature has focused on compressing the original weights or reducing the number of parameters required for fine-tuning these models. These compression methods generally constrain the parameter space, for example, through low-rank reparametrization (e.g., LoRA), pruning, or quantization (e.g., QLoRA) during or after the model training. In this paper, we present a novel model compression method, which we term Manifold-Constrained Neural Compression (MCNC). This method constrains the parameter space to low-dimensional pre-defined and frozen nonlinear manifolds,…
Peer Reviews
Decision·ICLR 2025 Poster
+ The proposed algorithm outperforms baselines in the high compression rate regions + The idea is interesting and shows that there exists a set of parameters in a DNN that lies in the lower-dimension manifold. + Evaluations are thorough
The paper presents an interesting idea of compressing neural network parameters into a low dimension manifold. Instead of training a generator to produce the neural network parameters, the proposed approach can use a randomly-initialized generator to project a trainable k-dimension vector into a d-dimension vector as parameters. I find the idea very interesting. The empirical results also demonstrate that the approach can achieve better accuracy under extreme compression rates. With that said,
The paper presents very simple yet powerful idea of essentially hashing high-dimensional weights into lower-dimensional manifold. Instead of hashing directly from higher to lower dimension, the proposed approach fixes the inverts of the hashing function (i.e., generator), and learns the "hashed" outputs via sgd. This idea has deep roots in random projections/random hashing literature, and as such I am very glad that similar approach can be exploited for the parameter efficient training of the ne
Overall, I think the paper does a good job in validating the soundness and effectiveness of the proposed approach, however, I think it can be significantly strengthened: 1. In the exposition of the idea, the actual architecture of the generator network comes very late; with actual numbers/settings coming in appendix! I think, the simplicity of the approach must be showcased from the beginning, I highly suggest to add simple pytorch code showing how easy is to plug and play with the method 2. Sim
1. Strong empirical results, with comparisons against other compression methods on various modalities and architectures demonstrating the proposed method’s effectiveness. 2. Extensive ablation studies for each component of the method, as well as for choices of hyper-parameters, giving some intuition on how 3. Demonstration of real world speedups
1. My main gripe with the paper is that the method is not very well motivated. For example, the paper claims that a randomly initialized generator is enough to span the entire space of weights in a higher dimension, but never shows much empirical evidence beyond an experiment in low dimensions. It is not clear if this result also holds in higher dimensions. I would have ideally liked to see a theoretical result around the expressivity of the manifold constrained reparametrization, but even an em
Videos
Taxonomy
TopicsAdvanced Data Compression Techniques · Video Coding and Compression Technologies · Graph Theory and Algorithms