Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity

TL;DR

This paper investigates how nonlinear dendrites in neural networks can expand capacity and reduce communication costs, offering insights for designing more efficient neural network hardware.

Contribution

It systematically analyzes the role of dendritic nonlinearities, revealing their primary benefit in capacity expansion and communication cost reduction rather than learning enhancement.

Findings

Dendritic nonlinearities do not significantly improve learning capacity.

They enable network capacity expansion with minimal communication overhead.

Implications for designing neural network accelerators to reduce training and inference costs.

Abstract

Our understanding of biological neuronal networks has profoundly influenced the development of artificial neural networks (ANNs). However, neurons utilized in ANNs differ considerably from their biological counterparts, primarily due to the absence of complex dendritic trees with local nonlinearities. Early studies have suggested that dendritic nonlinearities could substantially improve the learning capabilities of neural network models. In this study, we systematically examined the role of nonlinear dendrites within neural networks. Utilizing machine-learning methodologies, we assessed how dendritic nonlinearities influence neural network performance. Our findings demonstrate that dendritic nonlinearities do not substantially affect learning capacity; rather, their primary benefit lies in enabling network capacity expansion while minimizing communication costs through effective localized feature aggregation. This research provides critical insights with significant implications for designing future neural network accelerators aimed at reducing communication overhead during neural network training and inference.