Parameter Efficient Deep Neural Networks with Bilinear Projections

TL;DR

This paper introduces bilinear projections in deep neural networks to significantly reduce parameter count and model size, while maintaining or improving accuracy, making models more suitable for memory-limited devices.

Contribution

It proposes a novel bilinear projection method that reduces model complexity from quadratic to linear, addressing parameter redundancy and enabling efficient deep models.

Findings

Achieves higher accuracy than full DNNs on benchmarks

Reduces model size significantly

Maintains or improves accuracy with fewer parameters

Abstract

Recent research on deep neural networks (DNNs) has primarily focused on improving the model accuracy. Given a proper deep learning framework, it is generally possible to increase the depth or layer width to achieve a higher level of accuracy. However, the huge number of model parameters imposes more computational and memory usage overhead and leads to the parameter redundancy. In this paper, we address the parameter redundancy problem in DNNs by replacing conventional full projections with bilinear projections. For a fully-connected layer with $D$ input nodes and $D$ output nodes, applying bilinear projection can reduce the model space complexity from $\mathcal{O}(D^2)$ to $\mathcal{O}(2D)$, achieving a deep model with a sub-linear layer size. However, structured projection has a lower freedom of degree compared to the full projection, causing the under-fitting problem. So we simply scale up the mapping size by increasing the number of output channels, which can keep and even boosts the model accuracy. This makes it very parameter-efficient and handy to deploy such deep models on mobile systems with memory limitations. Experiments on four benchmark datasets show that applying the proposed bilinear projection to deep neural networks can achieve even higher accuracies than conventional full DNNs, while significantly reduces the model size.