Low-Resolution Neural Networks

TL;DR

This paper investigates the effects of reducing parameter bit precision in neural networks, demonstrating that low-resolution models can maintain performance with significant memory savings, especially when using 2.32-bit weights.

Contribution

It provides a comprehensive analysis of low-resolution neural networks across various architectures, highlighting the potential for memory-efficient models with minimal performance loss.

Findings

Low-resolution models with 2.32-bit weights perform comparably to 32-bit models.

Models with fewer parameters need more epochs to reach similar accuracy.

Including zero as a weight value stabilizes training in low-resolution models.

Abstract

The expanding scale of large neural network models introduces significant challenges, driving efforts to reduce memory usage and enhance computational efficiency. Such measures are crucial to ensure the practical implementation and effective application of these sophisticated models across a wide array of use cases. This study examines the impact of parameter bit precision on model performance compared to standard 32-bit models, with a focus on multiclass object classification in images. The models analyzed include those with fully connected layers, convolutional layers, and transformer blocks, with model weight resolution ranging from 1 bit to 4.08 bits. The findings indicate that models with lower parameter bit precision achieve results comparable to 32-bit models, showing promise for use in memory-constrained devices. While low-resolution models with a small number of parameters require more training epochs to achieve accuracy comparable to 32-bit models, those with a large number of parameters achieve similar performance within the same number of epochs. Additionally, data augmentation can destabilize training in low-resolution models, but including zero as a potential value in the weight parameters helps maintain stability and prevents performance degradation. Overall, 2.32-bit weights offer the optimal balance of memory reduction, performance, and efficiency. However, further research should explore other dataset types and more complex and larger models. These findings suggest a potential new era for optimized neural network models with reduced memory requirements and improved computational efficiency, though advancements in dedicated hardware are necessary to fully realize this potential.