Diversity Networks: Neural Network Compression Using Determinantal Point Processes

TL;DR

Divnet introduces a novel neural network compression method that employs Determinantal Point Processes to select diverse neurons, enabling effective pruning and size reduction while maintaining performance.

Contribution

It presents a new, principled approach using DPPs for neuron selection and fusion, improving network pruning and regularization over prior methods.

Findings

Divnet outperforms existing pruning methods in experiments.

It achieves smaller networks without performance loss.

Divnet is compatible with other memory reduction techniques.

Abstract

We introduce Divnet, a flexible technique for learning networks with diverse neurons. Divnet models neuronal diversity by placing a Determinantal Point Process (DPP) over neurons in a given layer. It uses this DPP to select a subset of diverse neurons and subsequently fuses the redundant neurons into the selected ones. Compared with previous approaches, Divnet offers a more principled, flexible technique for capturing neuronal diversity and thus implicitly enforcing regularization. This enables effective auto-tuning of network architecture and leads to smaller network sizes without hurting performance. Moreover, through its focus on diversity and neuron fusing, Divnet remains compatible with other procedures that seek to reduce memory footprints of networks. We present experimental results to corroborate our claims: for pruning neural networks, Divnet is seen to be notably superior to competing approaches.