Neural Networks Reduction via Lumping

TL;DR

This paper introduces a novel neural network reduction method based on lumpability, enabling neuron pruning without data or fine-tuning while preserving exact behavior, bridging network compression and formal verification techniques.

Contribution

It proposes a formal lumpability-based pruning approach that reduces neurons without retraining, connecting network compression with Markov chain verification concepts.

Findings

Reduces network size without data or fine-tuning.

Preserves exact network behavior after reduction.

Provides formal explanation for common reduction techniques.

Abstract

The increasing size of recently proposed Neural Networks makes it hard to implement them on embedded devices, where memory, battery and computational power are a non-trivial bottleneck. For this reason during the last years network compression literature has been thriving and a large number of solutions has been been published to reduce both the number of operations and the parameters involved with the models. Unfortunately, most of these reducing techniques are actually heuristic methods and usually require at least one re-training step to recover the accuracy. The need of procedures for model reduction is well-known also in the fields of Verification and Performances Evaluation, where large efforts have been devoted to the definition of quotients that preserve the observable underlying behaviour. In this paper we try to bridge the gap between the most popular and very effective network reduction strategies and formal notions, such as lumpability, introduced for verification and evaluation of Markov Chains. Elaborating on lumpability we propose a pruning approach that reduces the number of neurons in a network without using any data or fine-tuning, while completely preserving the exact behaviour. Relaxing the constraints on the exact definition of the quotienting method we can give a formal explanation of some of the most common reduction techniques.