Two-Valued Symmetric Circulant Matrices: Applications in Deep Learning

TL;DR

This paper introduces the Two-Valued Symmetric Circulant Matrix (TVSCM), a highly sparse neural network architecture that drastically reduces model size and power consumption, suitable for resource-limited edge devices.

Contribution

The paper proposes TVSCM, a novel sparse matrix architecture that uses only two weights per layer, significantly reducing storage and computational requirements for deep learning models.

Findings

Over 80x reduction in model parameters on MNIST and MIT-BIH datasets.

Maintains high accuracy with minimal parameters, e.g., 97.6% to 93.5% on MNIST.

Suitable for edge and IoMT devices due to low power and storage needs.

Abstract

Despite the success of deep neural networks in vision, medical diagnosis, and IoT scenarios, their deployment on resource-limited platforms poses serious challenges due to their high storage requirements, computational complexity, and large footprint. In particular, fully connected layers require a large number of weights, making it difficult for edge devices to accommodate them. To overcome these challenges associated with limited platforms, this paper proposes the Two-Valued Symmetric Circulant Matrix (TVSCM), a very sparse architecture that employs just two weights per layer to keep it circulant and symmetric. The extreme form of structured sparse architecture provides negligible storage costs compared to traditional full-weight storage. Instead of hardware and additional stages of other traditional sparse learning techniques, such as low-rank approximation and pruning approaches, this architecture provides an extreme form of sparsity, achieving very minimal storage requirements. The simulation study demonstrates more than 80$\times$ reduction in model parameters, reducing parameters from 623,290 to 7,852 on MNIST and from 24,709 to 942 on the MIT-BIH arrhythmia dataset, while maintaining comparable accuracy from 97.6% to 93.5% on MNIST and from 97.6% to 93.1% on MIT-BIH. Due to its minimal architectural requirements and very low power consumption, this architecture would be ideal for edge computing platforms, tiny-ML platforms, IoMT systems, and battery-powered systems.