Element-wise Modulation of Random Matrices for Efficient Neural Layers

TL;DR

This paper introduces the PRP layer, a new neural network component that uses element-wise modulation of fixed random matrices to significantly reduce parameters while maintaining accuracy, enabling efficient deployment in resource-constrained environments.

Contribution

The paper presents the Parametrized Random Projection (PRP) layer, a novel method that decouples feature mixing from adaptation, reducing parameters and computational cost without sacrificing performance.

Findings

Reduces trainable parameters to a linear scale.

Maintains reliable accuracy across various benchmarks.

Provides a stable, efficient solution for resource-limited deployment.

Abstract

Fully connected layers are a primary source of memory and computational overhead in deep neural networks due to their dense, often redundant parameterization. While various compression techniques exist, they frequently introduce complex engineering trade-offs or degrade model performance. We propose the Parametrized Random Projection (PRP) layer, a novel approach that decouples feature mixing from adaptation by utilizing a fixed random matrix modulated by lightweight, learnable element-wise parameters. This architecture drastically reduces the trainable parameter count to a linear scale while retaining reliable accuracy across various benchmarks. The design serves as a stable, computationally efficient solution for architectural scaling and deployment in resource-limited settings.