Synthesis and Pruning as a Dynamic Compression Strategy for Efficient Deep Neural Networks

TL;DR

This paper introduces a brain-inspired synthesis and pruning strategy for deep neural networks that enhances compression by selectively rewiring and pruning network connections, leading to smaller, efficient models.

Contribution

It presents a novel synthesis algorithm inspired by brain behavior, combined with pruning, to produce highly compressed and efficient deep neural networks.

Findings

Residual sub-networks share up to 90% similarity.

Combining synthesis with pruning improves compression.

The approach effectively reduces network size while maintaining performance.

Abstract

The brain is a highly reconfigurable machine capable of task-specific adaptations. The brain continually rewires itself for a more optimal configuration to solve problems. We propose a novel strategic synthesis algorithm for feedforward networks that draws directly from the brain's behaviours when learning. The proposed approach analyses the network and ranks weights based on their magnitude. Unlike existing approaches that advocate random selection, we select highly performing nodes as starting points for new edges and exploit the Gaussian distribution over the weights to select corresponding endpoints. The strategy aims only to produce useful connections and result in a smaller residual network structure. The approach is complemented with pruning to further the compression. We demonstrate the techniques to deep feedforward networks. The residual sub-networks that are formed from the synthesis approaches in this work form common sub-networks with similarities up to ~90%. Using pruning as a complement to the strategic synthesis approach, we observe improvements in compression.