Energy-Efficient Information Representation in MNIST Classification Using Biologically Inspired Learning

TL;DR

This paper presents a biologically inspired learning rule for MNIST classification that reduces overparameterization, improves efficiency, and enhances adaptability, addressing energy consumption and scalability issues in neural networks.

Contribution

It introduces a novel learning rule emulating brain plasticity that prevents overparameterization and outperforms backpropagation in efficiency and storage capacity.

Findings

Reduces network redundancy by optimizing synaptic usage.

Outperforms backpropagation in efficiency and capacity.

Eliminates the need for pre-optimized architectures.

Abstract

Efficient representation learning is essential for optimal information storage and classification. However, it is frequently overlooked in artificial neural networks (ANNs). This neglect results in networks that can become overparameterized by factors of up to 13, increasing redundancy and energy consumption. As the demand for large language models (LLMs) and their scale increase, these issues are further highlighted, raising significant ethical and environmental concerns. We analyze our previously developed biologically inspired learning rule using information-theoretic concepts, evaluating its efficiency on the MNIST classification task. The proposed rule, which emulates the brain's structural plasticity, naturally prevents overparameterization by optimizing synaptic usage and retaining only the essential number of synapses. Furthermore, it outperforms backpropagation (BP) in terms of efficiency and storage capacity. It also eliminates the need for pre-optimization of network architecture, enhances adaptability, and reflects the brain's ability to reserve 'space' for new memories. This approach advances scalable and energy-efficient AI and provides a promising framework for developing brain-inspired models that optimize resource allocation and adaptability.