Dendritic Integration Based Quadratic Neural Networks Outperform Traditional Aritificial Ones

TL;DR

This paper introduces a brain-inspired quadratic neural network model that outperforms traditional ANNs in classification tasks, with theoretical analysis of its generalization capabilities and a low-rank variant to reduce computational costs.

Contribution

The paper proposes the DIQNN model based on dendritic quadratic integration, along with a low-rank version and a theoretical generalization margin analysis.

Findings

DIQNN outperforms traditional ANNs in classification accuracy.

Low-rank DIQNN retains performance while reducing computational cost.

The generalization margin increases monotonically during training, correlating with test accuracy improvements.

Abstract

Incorporating biological neuronal properties into Artificial Neural Networks (ANNs) to enhance computational capabilities poses a formidable challenge in the field of machine learning. Inspired by recent findings indicating that dendrites adhere to quadratic integration rules for synaptic inputs, we propose a novel ANN model, Dendritic Integration-Based Quadratic Neural Network (DIQNN). This model shows superior performance over traditional ANNs in a variety of classification tasks. To reduce the computational cost of DIQNN, we introduce the Low-Rank DIQNN, while we find it can retain the performance of the original DIQNN. We further propose a margin to characterize the generalization error and theoretically prove this margin will increase monotonically during training. And we show the consistency between generalization and our margin using numerical experiments. Finally, by integrating this margin into the loss function, the change of test accuracy is indeed accelerated. Our work contributes a novel, brain-inspired ANN model that surpasses traditional ANNs and provides a theoretical framework to analyze the generalization error in classification tasks.